Finetune Gemma

Wed Feb 21 17:19:05 2024       
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 545.23.08              Driver Version: 545.23.08    CUDA Version: 12.3     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA A100 80GB PCIe          On  | 00000001:00:00.0 Off |                    0 |
| N/A   33C    P0              42W / 300W |      4MiB / 81920MiB |      0%      Default |
|                                         |                      |             Disabled |
+-----------------------------------------+----------------------+----------------------+
                                                                                         
+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|  No running processes found                                                           |
+---------------------------------------------------------------------------------------+

A decoder-only architecture is being used, but right-padding was detected! For correct generation results, please set `padding_side='left'` when initializing the tokenizer.

  user
  Below is an instruction that describes a task. Write a response that appropriately completes the request.
  code the fibonacci series in python using reccursion
  
model
  

   a Python function to calculate the nth Fibonacci number using recursion. Here's the code: 

```python

def fibonacci(n):
  if n == 0:
    return 0
  elif n == 1:
    return 1
  else:
    return fibonacci(n-1) + fibonacci(n-2)

# Print the nth Fibonacci number
print(fibonacci(10))
```

**Explanation:**

1. The function `fibonacci` takes an integer `n` as input.
2. If `n` is 0 or 1, it returns the respective base case of 0 or 1.
3. Otherwise, it recursively calculates the Fibonacci number for `n-1` and `n-2` and adds their sum to return the Fibonacci number for `n`.
4. The function continues to recurse until `n` reaches the desired number, and the final result is returned.

**Output:**

```
>>> print(fibonacci(10))
5
```

In this example, the code calculates the 10th Fibonacci number, which is 5.

**Note:** Reccursion can be a powerful technique for solving problems that involve repeated calculations. However, it is important to note that recursion can also lead to stack overflow errors for large values of `n` due to its repeated function calls. For more efficient solutions, iterative approaches are often preferred.

Dataset({
    features: ['input', 'output', 'text', 'instruction', 'prompt', 'input_ids', 'attention_mask'],
    num_rows: 24392
})

GemmaForCausalLM(
  (model): GemmaModel(
    (embed_tokens): Embedding(256000, 3072, padding_idx=0)
    (layers): ModuleList(
      (0-27): 28 x GemmaDecoderLayer(
        (self_attn): GemmaSdpaAttention(
          (q_proj): Linear4bit(in_features=3072, out_features=4096, bias=False)
          (k_proj): Linear4bit(in_features=3072, out_features=4096, bias=False)
          (v_proj): Linear4bit(in_features=3072, out_features=4096, bias=False)
          (o_proj): Linear4bit(in_features=4096, out_features=3072, bias=False)
          (rotary_emb): GemmaRotaryEmbedding()
        )
        (mlp): GemmaMLP(
          (gate_proj): Linear4bit(in_features=3072, out_features=24576, bias=False)
          (up_proj): Linear4bit(in_features=3072, out_features=24576, bias=False)
          (down_proj): Linear4bit(in_features=24576, out_features=3072, bias=False)
          (act_fn): GELUActivation()
        )
        (input_layernorm): GemmaRMSNorm()
        (post_attention_layernorm): GemmaRMSNorm()
      )
    )
    (norm): GemmaRMSNorm()
  )
  (lm_head): Linear(in_features=3072, out_features=256000, bias=False)
)

['o_proj', 'q_proj', 'up_proj', 'v_proj', 'k_proj', 'down_proj', 'gate_proj']

Trainable: 200015872 | total: 8737696768 | Percentage: 2.2891%

/home/adithya/miniconda3/envs/gemma-venv/lib/python3.10/site-packages/torch/utils/checkpoint.py:460: UserWarning: torch.utils.checkpoint: please pass in use_reentrant=True or use_reentrant=False explicitly. The default value of use_reentrant will be updated to be False in the future. To maintain current behavior, pass use_reentrant=True. It is recommended that you use use_reentrant=False. Refer to docs for more details on the differences between the two variants.
  warnings.warn(

A decoder-only architecture is being used, but right-padding was detected! For correct generation results, please set `padding_side='left'` when initializing the tokenizer.

  user
  Below is an instruction that describes a task. Write a response that appropriately completes the request.
  code the fibonacci series in python using reccursion
  
model
  

   a Python program to code the Fibonacci series using recursion. Here's a solution:

```python
def fibonacci(n):
  """Calculates the nth Fibonacci number using recursion.

  Args:
    n: The index of the Fibonacci number to calculate.

  Returns:
    The nth Fibonacci number.
  """

  # Base case: The first two Fibonacci numbers are 0 and 1.
  if n <= 1:
    return n

  # Recursive case: Otherwise, calculate the nth Fibonacci number by adding the previous two numbers.
  else:
    return fibonacci(n-1) + fibonacci(n-2)


# Print the Fibonacci numbers.
for i in range(10):
  print(fibonacci(i))
```

**Explanation:**

* The `fibonacci` function takes an integer `n` as input.
* If `n` is less than or equal to 1, it returns `n` itself, as the base case.
* Otherwise, it calculates `n`-th Fibonacci number recursively by adding the previous two numbers.
* The function calls itself with smaller and smaller values of `n` until it reaches the base case.

**Example Usage:**

```python
# Print the Fibonacci numbers from 0 to 10.
for i in range(10):
  print(fibonacci(i))
```

**Output:**

```
0
1
1
2
3
5
8
13
21
34
```

**Note:**

* This program calculates Fibonacci numbers recursively, which means that it may not be very efficient for large values of `n` as it repeats calculations.
* For more efficient Fibonacci number calculation, consider using an iterative approach rather than recursion.