{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Fine-tuning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the previous section, we went through how to construct training and testing data properly. In this tutorial, we will actually fine-tune the model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Installation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note to fine-tune BGE models using FlagEmbedding, we need to install the package with the finetune dependency:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "% pip install -U FlagEmbedding[finetune]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Fine-tune"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Below are the arguments for fine-tuning:\n",
    "\n",
    "The following arguments are for model:\n",
    "- `model_name_or_path`: The model checkpoint for initialization.\n",
    "- `config_name`: Pretrained config name or path if not the same as model_name.\n",
    "- `tokenizer_name`: Pretrained tokenizer name or path if not the same as model_name.\n",
    "- `cache_dir`: Where do you want to store the pre-trained models downloaded from s3.\n",
    "- `trust_remote_code`: Trust remote code\n",
    "- `token`: The token to use when accessing the model.\n",
    "\n",
    "The following arguments are for data:\n",
    "- `train_data`: One or more paths to training data. `query: str`, `pos: List[str]`, `neg: List[str]` are required in the training data. Argument type: multiple.\n",
    "- `cache_path`: Where do you want to store the cached data.\n",
    "- `train_group_size`: (No metadata provided)\n",
    "- `query_max_len`: The maximum total input sequence length after tokenization for passage. Sequences longer than this will be truncated.\n",
    "- `passage_max_len`: The maximum total input sequence length after tokenization for passage. Sequences longer than this will be truncated.\n",
    "- `pad_to_multiple_of`: If set will pad the sequence to be a multiple of the provided value.\n",
    "- `max_example_num_per_dataset`: The max number of examples for each dataset.\n",
    "- `query_instruction_for_retrieval`: Instruction for query.\n",
    "- `query_instruction_format`: Format for query instruction.\n",
    "- `knowledge_distillation`: Use knowledge distillation when `pos_scores: List[float]` and `neg_scores: List[float]` are in features of training data.\n",
    "- `passage_instruction_for_retrieval`: Instruction for passage.\n",
    "- `passage_instruction_format`: Format for passage instruction.\n",
    "- `shuffle_ratio`: The ratio of shuffling the text.\n",
    "- `same_dataset_within_batch`: All samples in the same batch comes from the same dataset.\n",
    "- `small_threshold`: The threshold of small dataset. All small dataset in the same directory will be merged into one dataset.\n",
    "- `drop_threshold`: The threshold for dropping merged small dataset. If the number of examples in the merged small dataset is less than this threshold, it will be dropped.\n",
    "\n",
    "And the following extra arguments:\n",
    "- `negatives_cross_device`: Share negatives across devices.\n",
    "- `temperature`: Temperature used for similarity score.\n",
    "- `fix_position_embedding`: Freeze the parameters of position embeddings.\n",
    "- `sentence_pooling_method`: The pooling method. Available options: cls, mean, last_token. Default: cls.\n",
    "- `normalize_embeddings`: Whether to normalize the embeddings.\n",
    "- `sub_batch_size`: Sub batch size for training.\n",
    "- `kd_loss_type`: The loss type for knowledge distillation. Available options: kl_div, m3_kd_loss. Default: kl_div."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "W1223 06:27:06.807000 1362426 site-packages/torch/distributed/run.py:793] \n",
      "W1223 06:27:06.807000 1362426 site-packages/torch/distributed/run.py:793] *****************************************\n",
      "W1223 06:27:06.807000 1362426 site-packages/torch/distributed/run.py:793] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. \n",
      "W1223 06:27:06.807000 1362426 site-packages/torch/distributed/run.py:793] *****************************************\n",
      "/share/project/xzy/Envs/ft/lib/python3.11/site-packages/_distutils_hack/__init__.py:54: UserWarning: Reliance on distutils from stdlib is deprecated. Users must rely on setuptools to provide the distutils module. Avoid importing distutils or import setuptools first, and avoid setting SETUPTOOLS_USE_DISTUTILS=stdlib. Register concerns at https://github.com/pypa/setuptools/issues/new?template=distutils-deprecation.yml\n",
      "  warnings.warn(\n",
      "/share/project/xzy/Envs/ft/lib/python3.11/site-packages/_distutils_hack/__init__.py:54: UserWarning: Reliance on distutils from stdlib is deprecated. Users must rely on setuptools to provide the distutils module. Avoid importing distutils or import setuptools first, and avoid setting SETUPTOOLS_USE_DISTUTILS=stdlib. Register concerns at https://github.com/pypa/setuptools/issues/new?template=distutils-deprecation.yml\n",
      "  warnings.warn(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[2024-12-23 06:27:31,423] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)\n",
      "[2024-12-23 06:27:31,424] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)\n",
      "[2024-12-23 06:27:40,529] [INFO] [comm.py:652:init_distributed] cdb=None\n",
      "[2024-12-23 06:27:40,529] [INFO] [comm.py:652:init_distributed] cdb=None\n",
      "[2024-12-23 06:27:40,529] [INFO] [comm.py:683:init_distributed] Initializing TorchBackend in DeepSpeed with backend nccl\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "12/23/2024 06:27:40 - WARNING - FlagEmbedding.abc.finetune.embedder.AbsRunner -   Process rank: 0, device: cuda:0, n_gpu: 1, distributed training: True, 16-bits training: True\n",
      "12/23/2024 06:27:40 - INFO - FlagEmbedding.abc.finetune.embedder.AbsRunner -   Training/evaluation parameters AbsEmbedderTrainingArguments(\n",
      "_n_gpu=1,\n",
      "accelerator_config={'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None, 'use_configured_state': False},\n",
      "adafactor=False,\n",
      "adam_beta1=0.9,\n",
      "adam_beta2=0.999,\n",
      "adam_epsilon=1e-08,\n",
      "auto_find_batch_size=False,\n",
      "batch_eval_metrics=False,\n",
      "bf16=False,\n",
      "bf16_full_eval=False,\n",
      "data_seed=None,\n",
      "dataloader_drop_last=True,\n",
      "dataloader_num_workers=0,\n",
      "dataloader_persistent_workers=False,\n",
      "dataloader_pin_memory=True,\n",
      "dataloader_prefetch_factor=None,\n",
      "ddp_backend=None,\n",
      "ddp_broadcast_buffers=None,\n",
      "ddp_bucket_cap_mb=None,\n",
      "ddp_find_unused_parameters=None,\n",
      "ddp_timeout=1800,\n",
      "debug=[],\n",
      "deepspeed=config/ds_stage0.json,\n",
      "disable_tqdm=False,\n",
      "dispatch_batches=None,\n",
      "do_eval=False,\n",
      "do_predict=False,\n",
      "do_train=False,\n",
      "eval_accumulation_steps=None,\n",
      "eval_delay=0,\n",
      "eval_do_concat_batches=True,\n",
      "eval_on_start=False,\n",
      "eval_steps=None,\n",
      "eval_strategy=IntervalStrategy.NO,\n",
      "eval_use_gather_object=False,\n",
      "evaluation_strategy=None,\n",
      "fix_position_embedding=False,\n",
      "fp16=True,\n",
      "fp16_backend=auto,\n",
      "fp16_full_eval=False,\n",
      "fp16_opt_level=O1,\n",
      "fsdp=[],\n",
      "fsdp_config={'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False},\n",
      "fsdp_min_num_params=0,\n",
      "fsdp_transformer_layer_cls_to_wrap=None,\n",
      "full_determinism=False,\n",
      "gradient_accumulation_steps=1,\n",
      "gradient_checkpointing=True,\n",
      "gradient_checkpointing_kwargs=None,\n",
      "greater_is_better=None,\n",
      "group_by_length=False,\n",
      "half_precision_backend=auto,\n",
      "hub_always_push=False,\n",
      "hub_model_id=None,\n",
      "hub_private_repo=False,\n",
      "hub_strategy=HubStrategy.EVERY_SAVE,\n",
      "hub_token=<HUB_TOKEN>,\n",
      "ignore_data_skip=False,\n",
      "include_inputs_for_metrics=False,\n",
      "include_num_input_tokens_seen=False,\n",
      "include_tokens_per_second=False,\n",
      "jit_mode_eval=False,\n",
      "kd_loss_type=kl_div,\n",
      "label_names=None,\n",
      "label_smoothing_factor=0.0,\n",
      "learning_rate=1e-05,\n",
      "length_column_name=length,\n",
      "load_best_model_at_end=False,\n",
      "local_rank=0,\n",
      "log_level=passive,\n",
      "log_level_replica=warning,\n",
      "log_on_each_node=True,\n",
      "logging_dir=./test_encoder_only_base_bge-large-en-v1.5/runs/Dec23_06-27-30_job-40fb0ce3-8bfb-46ea-b409-0a2e2a1a3163-master-0,\n",
      "logging_first_step=False,\n",
      "logging_nan_inf_filter=True,\n",
      "logging_steps=1.0,\n",
      "logging_strategy=IntervalStrategy.STEPS,\n",
      "lr_scheduler_kwargs={},\n",
      "lr_scheduler_type=SchedulerType.LINEAR,\n",
      "max_grad_norm=1.0,\n",
      "max_steps=-1,\n",
      "metric_for_best_model=None,\n",
      "mp_parameters=,\n",
      "neftune_noise_alpha=None,\n",
      "negatives_cross_device=True,\n",
      "no_cuda=False,\n",
      "normalize_embeddings=True,\n",
      "num_train_epochs=2.0,\n",
      "optim=OptimizerNames.ADAMW_TORCH,\n",
      "optim_args=None,\n",
      "optim_target_modules=None,\n",
      "output_dir=./test_encoder_only_base_bge-large-en-v1.5,\n",
      "overwrite_output_dir=True,\n",
      "past_index=-1,\n",
      "per_device_eval_batch_size=8,\n",
      "per_device_train_batch_size=2,\n",
      "prediction_loss_only=False,\n",
      "push_to_hub=False,\n",
      "push_to_hub_model_id=None,\n",
      "push_to_hub_organization=None,\n",
      "push_to_hub_token=<PUSH_TO_HUB_TOKEN>,\n",
      "ray_scope=last,\n",
      "remove_unused_columns=True,\n",
      "report_to=[],\n",
      "restore_callback_states_from_checkpoint=False,\n",
      "resume_from_checkpoint=None,\n",
      "run_name=./test_encoder_only_base_bge-large-en-v1.5,\n",
      "save_on_each_node=False,\n",
      "save_only_model=False,\n",
      "save_safetensors=True,\n",
      "save_steps=1000,\n",
      "save_strategy=IntervalStrategy.STEPS,\n",
      "save_total_limit=None,\n",
      "seed=42,\n",
      "sentence_pooling_method=cls,\n",
      "skip_memory_metrics=True,\n",
      "split_batches=None,\n",
      "sub_batch_size=None,\n",
      "temperature=0.02,\n",
      "tf32=None,\n",
      "torch_compile=False,\n",
      "torch_compile_backend=None,\n",
      "torch_compile_mode=None,\n",
      "torch_empty_cache_steps=None,\n",
      "torchdynamo=None,\n",
      "tpu_metrics_debug=False,\n",
      "tpu_num_cores=None,\n",
      "use_cpu=False,\n",
      "use_ipex=False,\n",
      "use_legacy_prediction_loop=False,\n",
      "use_mps_device=False,\n",
      "warmup_ratio=0.1,\n",
      "warmup_steps=0,\n",
      "weight_decay=0.0,\n",
      ")\n",
      "12/23/2024 06:27:40 - INFO - FlagEmbedding.abc.finetune.embedder.AbsRunner -   Model parameters AbsEmbedderModelArguments(model_name_or_path='BAAI/bge-large-en-v1.5', config_name=None, tokenizer_name=None, cache_dir='./cache/model', trust_remote_code=False, token=None)\n",
      "12/23/2024 06:27:40 - INFO - FlagEmbedding.abc.finetune.embedder.AbsRunner -   Data parameters AbsEmbedderDataArguments(train_data=['./ft_data/training.json'], cache_path='./cache/data', train_group_size=8, query_max_len=512, passage_max_len=512, pad_to_multiple_of=8, max_example_num_per_dataset=100000000, query_instruction_for_retrieval='Represent this sentence for searching relevant passages: ', query_instruction_format='{}{}', knowledge_distillation=False, passage_instruction_for_retrieval=None, passage_instruction_format='{}{}', shuffle_ratio=0.0, same_dataset_within_batch=False, small_threshold=0, drop_threshold=0)\n",
      "12/23/2024 06:27:40 - WARNING - FlagEmbedding.abc.finetune.embedder.AbsRunner -   Process rank: 1, device: cuda:1, n_gpu: 1, distributed training: True, 16-bits training: True\n",
      "12/23/2024 06:35:01 - INFO - FlagEmbedding.finetune.embedder.encoder_only.base.runner -   Config: BertConfig {\n",
      "  \"_name_or_path\": \"BAAI/bge-large-en-v1.5\",\n",
      "  \"architectures\": [\n",
      "    \"BertModel\"\n",
      "  ],\n",
      "  \"attention_probs_dropout_prob\": 0.1,\n",
      "  \"classifier_dropout\": null,\n",
      "  \"gradient_checkpointing\": false,\n",
      "  \"hidden_act\": \"gelu\",\n",
      "  \"hidden_dropout_prob\": 0.1,\n",
      "  \"hidden_size\": 1024,\n",
      "  \"id2label\": {\n",
      "    \"0\": \"LABEL_0\"\n",
      "  },\n",
      "  \"initializer_range\": 0.02,\n",
      "  \"intermediate_size\": 4096,\n",
      "  \"label2id\": {\n",
      "    \"LABEL_0\": 0\n",
      "  },\n",
      "  \"layer_norm_eps\": 1e-12,\n",
      "  \"max_position_embeddings\": 512,\n",
      "  \"model_type\": \"bert\",\n",
      "  \"num_attention_heads\": 16,\n",
      "  \"num_hidden_layers\": 24,\n",
      "  \"pad_token_id\": 0,\n",
      "  \"position_embedding_type\": \"absolute\",\n",
      "  \"torch_dtype\": \"float32\",\n",
      "  \"transformers_version\": \"4.44.2\",\n",
      "  \"type_vocab_size\": 2,\n",
      "  \"use_cache\": true,\n",
      "  \"vocab_size\": 30522\n",
      "}\n",
      "\n",
      "12/23/2024 06:35:01 - INFO - FlagEmbedding.abc.finetune.embedder.AbsDataset -   loading data from ./ft_data/training.json ...\n",
      "Generating train split: 6300 examples [00:00, 46043.95 examples/s]\n",
      "/share/project/xzy/Envs/ft/lib/python3.11/site-packages/transformers/deepspeed.py:24: FutureWarning: transformers.deepspeed module is deprecated and will be removed in a future version. Please import deepspeed modules directly from transformers.integrations\n",
      "  warnings.warn(\n",
      "/share/project/xzy/Envs/ft/lib/python3.11/site-packages/transformers/deepspeed.py:24: FutureWarning: transformers.deepspeed module is deprecated and will be removed in a future version. Please import deepspeed modules directly from transformers.integrations\n",
      "  warnings.warn(\n",
      "12/23/2024 06:35:02 - WARNING - accelerate.utils.other -   Detected kernel version 5.4.0, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1734935704.354551] [job-40fb0ce3-8bfb-46ea-b409-0a2e2a1a3163-master-0:1362491:f]        vfs_fuse.c:281  UCX  ERROR inotify_add_watch(/tmp) failed: No space left on device\n",
      "[1734935704.383634] [job-40fb0ce3-8bfb-46ea-b409-0a2e2a1a3163-master-0:1362492:f]        vfs_fuse.c:281  UCX  ERROR inotify_add_watch(/tmp) failed: No space left on device\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using /root/.cache/torch_extensions/py311_cu124 as PyTorch extensions root...\n",
      "Using /root/.cache/torch_extensions/py311_cu124 as PyTorch extensions root...\n",
      "Detected CUDA files, patching ldflags\n",
      "Emitting ninja build file /root/.cache/torch_extensions/py311_cu124/fused_adam/build.ninja...\n",
      "/share/project/xzy/Envs/ft/lib/python3.11/site-packages/torch/utils/cpp_extension.py:1964: UserWarning: TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation. \n",
      "If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'].\n",
      "  warnings.warn(\n",
      "Building extension module fused_adam...\n",
      "Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ninja: no work to do.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Loading extension module fused_adam...\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Time to load fused_adam op: 1.1966907978057861 seconds\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Loading extension module fused_adam...\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Time to load fused_adam op: 1.2037739753723145 seconds\n",
      "[2024-12-23 06:35:06,883] [WARNING] [lr_schedules.py:683:get_lr] Attempting to get learning rate from scheduler before it has started\n",
      "[2024-12-23 06:35:06,888] [WARNING] [lr_schedules.py:683:get_lr] Attempting to get learning rate from scheduler before it has started\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "You're using a BertTokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.\n",
      "  0%|          | 0/3150 [00:00<?, ?it/s]/share/project/xzy/Envs/ft/lib/python3.11/site-packages/transformers/tokenization_utils_base.py:2888: UserWarning: `max_length` is ignored when `padding`=`True` and there is no truncation strategy. To pad to max length, use `padding='max_length'`.\n",
      "  warnings.warn(\n",
      "You're using a BertTokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.\n",
      "/share/project/xzy/Envs/ft/lib/python3.11/site-packages/transformers/tokenization_utils_base.py:2888: UserWarning: `max_length` is ignored when `padding`=`True` and there is no truncation strategy. To pad to max length, use `padding='max_length'`.\n",
      "  warnings.warn(\n",
      "/share/project/xzy/Envs/ft/lib/python3.11/site-packages/torch/_dynamo/eval_frame.py:632: UserWarning: torch.utils.checkpoint: the use_reentrant parameter should be passed explicitly. In version 2.5 we will raise an exception if use_reentrant is not passed. use_reentrant=False is recommended, but if you need to preserve the current default behavior, you can pass use_reentrant=True. Refer to docs for more details on the differences between the two variants.\n",
      "  return fn(*args, **kwargs)\n",
      "/share/project/xzy/Envs/ft/lib/python3.11/site-packages/torch/_dynamo/eval_frame.py:632: UserWarning: torch.utils.checkpoint: the use_reentrant parameter should be passed explicitly. In version 2.5 we will raise an exception if use_reentrant is not passed. use_reentrant=False is recommended, but if you need to preserve the current default behavior, you can pass use_reentrant=True. Refer to docs for more details on the differences between the two variants.\n",
      "  return fn(*args, **kwargs)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.0124, 'grad_norm': 1.0943871958089542, 'learning_rate': 0.0, 'epoch': 0.0}\n",
      "{'loss': 0.1189, 'grad_norm': 9.971958134471109, 'learning_rate': 1.2049342512977792e-06, 'epoch': 0.0}\n",
      "{'loss': 0.0067, 'grad_norm': 0.676847884003986, 'learning_rate': 1.9097756041415023e-06, 'epoch': 0.0}\n",
      "{'loss': 1.5215, 'grad_norm': 40.51544573089919, 'learning_rate': 2.4098685025955585e-06, 'epoch': 0.0}\n",
      "{'loss': 0.0111, 'grad_norm': 0.8537607081175989, 'learning_rate': 2.7977706905803826e-06, 'epoch': 0.0}\n",
      "{'loss': 0.0019, 'grad_norm': 0.1699944264536089, 'learning_rate': 3.1147098554392813e-06, 'epoch': 0.0}\n",
      "{'loss': 0.0003, 'grad_norm': 0.026271846378513198, 'learning_rate': 3.3826781011366144e-06, 'epoch': 0.0}\n",
      "{'loss': 0.0039, 'grad_norm': 0.3161338881928349, 'learning_rate': 3.614802753893337e-06, 'epoch': 0.01}\n",
      "{'loss': 0.0351, 'grad_norm': 2.335078256835444, 'learning_rate': 3.8195512082830046e-06, 'epoch': 0.01}\n",
      "{'loss': 0.1005, 'grad_norm': 10.32570731855295, 'learning_rate': 4.002704941878162e-06, 'epoch': 0.01}\n",
      "{'loss': 0.0412, 'grad_norm': 5.065856950874997, 'learning_rate': 4.1683876473185966e-06, 'epoch': 0.01}\n",
      "{'loss': 0.0228, 'grad_norm': 1.4469018394007689, 'learning_rate': 4.319644106737061e-06, 'epoch': 0.01}\n",
      "{'loss': 0.4375, 'grad_norm': 25.32025705794609, 'learning_rate': 4.458786561250902e-06, 'epoch': 0.01}\n",
      "{'loss': 0.0677, 'grad_norm': 6.604963701770736, 'learning_rate': 4.587612352434394e-06, 'epoch': 0.01}\n",
      "{'loss': 0.0749, 'grad_norm': 6.64658251837619, 'learning_rate': 4.7075462947218845e-06, 'epoch': 0.01}\n",
      "{'loss': 0.0012, 'grad_norm': 0.08107203275012109, 'learning_rate': 4.819737005191117e-06, 'epoch': 0.01}\n",
      "{'loss': 0.0047, 'grad_norm': 0.5208537542790276, 'learning_rate': 4.925123978329471e-06, 'epoch': 0.01}\n",
      "{'loss': 0.0125, 'grad_norm': 1.2911095750936001, 'learning_rate': 5.024485459580783e-06, 'epoch': 0.01}\n",
      "{'loss': 0.0001, 'grad_norm': 0.012096519313285625, 'learning_rate': 5.118473357978383e-06, 'epoch': 0.01}\n",
      "{'loss': 0.0022, 'grad_norm': 0.2015078341112431, 'learning_rate': 5.207639193175941e-06, 'epoch': 0.01}\n",
      "{'loss': 0.0015, 'grad_norm': 0.1712907430029178, 'learning_rate': 5.292453705278116e-06, 'epoch': 0.01}\n",
      "{'loss': 0.1081, 'grad_norm': 9.019625564551282, 'learning_rate': 5.373321898616376e-06, 'epoch': 0.01}\n",
      "{'loss': 0.7266, 'grad_norm': 24.500381155357818, 'learning_rate': 5.450594738720674e-06, 'epoch': 0.01}\n",
      "{'loss': 0.4248, 'grad_norm': 20.19468915261394, 'learning_rate': 5.52457835803484e-06, 'epoch': 0.02}\n",
      "{'loss': 0.0003, 'grad_norm': 0.029853964107490208, 'learning_rate': 5.595541381160765e-06, 'epoch': 0.02}\n",
      "{'loss': 0.0261, 'grad_norm': 2.4565057480626606, 'learning_rate': 5.663720812548681e-06, 'epoch': 0.02}\n",
      "{'loss': 0.0002, 'grad_norm': 0.02366100843019717, 'learning_rate': 5.7293268124245064e-06, 'epoch': 0.02}\n",
      "{'loss': 0.0002, 'grad_norm': 0.011689008034651378, 'learning_rate': 5.792546603732173e-06, 'epoch': 0.02}\n",
      "{'loss': 0.0134, 'grad_norm': 1.6272253058407824, 'learning_rate': 5.853547693182881e-06, 'epoch': 0.02}\n",
      "{'loss': 0.0001, 'grad_norm': 0.006106387893044423, 'learning_rate': 5.912480546019664e-06, 'epoch': 0.02}\n",
      "{'loss': 0.0011, 'grad_norm': 0.11557933527457331, 'learning_rate': 5.9694808220382294e-06, 'epoch': 0.02}\n",
      "{'loss': 0.001, 'grad_norm': 0.06355657303327857, 'learning_rate': 6.0246712564888964e-06, 'epoch': 0.02}\n",
      "{'loss': 0.0004, 'grad_norm': 0.04231412288564197, 'learning_rate': 6.0781632514600984e-06, 'epoch': 0.02}\n",
      "{'loss': 0.0095, 'grad_norm': 0.9815642175717099, 'learning_rate': 6.130058229627251e-06, 'epoch': 0.02}\n",
      "{'loss': 0.1704, 'grad_norm': 16.072662344910864, 'learning_rate': 6.180448791716996e-06, 'epoch': 0.02}\n",
      "{'loss': 0.0175, 'grad_norm': 1.7137070627210673, 'learning_rate': 6.229419710878563e-06, 'epoch': 0.02}\n",
      "{'loss': 0.0226, 'grad_norm': 1.8205994473990903, 'learning_rate': 6.2770487907848145e-06, 'epoch': 0.02}\n",
      "{'loss': 0.0215, 'grad_norm': 2.1652719581476316, 'learning_rate': 6.323407609276162e-06, 'epoch': 0.02}\n",
      "{'loss': 0.005, 'grad_norm': 0.4481776405019213, 'learning_rate': 6.3685621653924034e-06, 'epoch': 0.02}\n",
      "{'loss': 0.0002, 'grad_norm': 0.012911016891643791, 'learning_rate': 6.41257344447372e-06, 'epoch': 0.03}\n",
      "{'loss': 0.0039, 'grad_norm': 0.36362350912565455, 'learning_rate': 6.455497913473407e-06, 'epoch': 0.03}\n",
      "{'loss': 0.6172, 'grad_norm': 17.30483211000525, 'learning_rate': 6.497387956575896e-06, 'epoch': 0.03}\n",
      "{'loss': 0.0012, 'grad_norm': 0.12979219617041665, 'learning_rate': 6.538292259550499e-06, 'epoch': 0.03}\n",
      "{'loss': 0.0001, 'grad_norm': 0.009205114279819068, 'learning_rate': 6.578256149914154e-06, 'epoch': 0.03}\n",
      "{'loss': 0.0, 'grad_norm': 0.00013248260316463386, 'learning_rate': 6.617321898863387e-06, 'epoch': 0.03}\n",
      "{'loss': 0.0003, 'grad_norm': 0.028528992331809232, 'learning_rate': 6.655528990018454e-06, 'epoch': 0.03}\n",
      "{'loss': 0.4529, 'grad_norm': 33.029949122917216, 'learning_rate': 6.692914359263185e-06, 'epoch': 0.03}\n",
      "{'loss': 0.1495, 'grad_norm': 12.704070296547062, 'learning_rate': 6.729512609332619e-06, 'epoch': 0.03}\n",
      "{'loss': 0.0001, 'grad_norm': 0.009230899786651852, 'learning_rate': 6.765356202273229e-06, 'epoch': 0.03}\n",
      "{'loss': 0.0058, 'grad_norm': 0.7164287401369475, 'learning_rate': 6.800475632458544e-06, 'epoch': 0.03}\n",
      "{'loss': 0.0301, 'grad_norm': 2.9980039313903237, 'learning_rate': 6.834899582470973e-06, 'epoch': 0.03}\n",
      "{'loss': 0.0174, 'grad_norm': 1.3938414847280698, 'learning_rate': 6.868655063846461e-06, 'epoch': 0.03}\n",
      "{'loss': 0.0483, 'grad_norm': 4.752412885883868, 'learning_rate': 6.901767544412343e-06, 'epoch': 0.03}\n",
      "{'loss': 0.0003, 'grad_norm': 0.03491949623495416, 'learning_rate': 6.934261063722286e-06, 'epoch': 0.03}\n",
      "{'loss': 0.0001, 'grad_norm': 0.006442951125193213, 'learning_rate': 6.966158337898979e-06, 'epoch': 0.03}\n",
      "{'loss': 0.0, 'grad_norm': 0.0006299447267779463, 'learning_rate': 6.997480855029952e-06, 'epoch': 0.04}\n",
      "{'loss': 0.0004, 'grad_norm': 0.04575714216250935, 'learning_rate': 7.028248962119886e-06, 'epoch': 0.04}\n",
      "{'loss': 0.0037, 'grad_norm': 0.3667356673614418, 'learning_rate': 7.058481944480661e-06, 'epoch': 0.04}\n",
      "{'loss': 0.1401, 'grad_norm': 13.978590807966391, 'learning_rate': 7.0881980983348955e-06, 'epoch': 0.04}\n",
      "{'loss': 0.0082, 'grad_norm': 0.773411220527937, 'learning_rate': 7.1174147973174426e-06, 'epoch': 0.04}\n",
      "{'loss': 0.0004, 'grad_norm': 0.037220675051924224, 'learning_rate': 7.1461485534801215e-06, 'epoch': 0.04}\n",
      "{'loss': 0.0172, 'grad_norm': 3.0044581349033983, 'learning_rate': 7.174415073336009e-06, 'epoch': 0.04}\n",
      "{'loss': 0.0, 'grad_norm': 0.0009021660741710128, 'learning_rate': 7.202229309419618e-06, 'epoch': 0.04}\n",
      "{'loss': 0.0003, 'grad_norm': 0.03098905288084688, 'learning_rate': 7.229605507786674e-06, 'epoch': 0.04}\n",
      "{'loss': 0.1153, 'grad_norm': 10.66327299332507, 'learning_rate': 7.256557251831284e-06, 'epoch': 0.04}\n",
      "{'loss': 0.0, 'grad_norm': 8.658231716040802e-05, 'learning_rate': 7.283097502757876e-06, 'epoch': 0.04}\n",
      "{'loss': 0.0021, 'grad_norm': 0.276288806597498, 'learning_rate': 7.309238637009787e-06, 'epoch': 0.04}\n",
      "{'loss': 0.0003, 'grad_norm': 0.029148684664189305, 'learning_rate': 7.334992480925029e-06, 'epoch': 0.04}\n",
      "{'loss': 0.0, 'grad_norm': 1.6249292676188826e-05, 'learning_rate': 7.360370342862176e-06, 'epoch': 0.04}\n",
      "{'loss': 0.0, 'grad_norm': 0.00022599051350145614, 'learning_rate': 7.3853830430147765e-06, 'epoch': 0.04}\n",
      "{'loss': 0.0039, 'grad_norm': 0.41817978852094234, 'learning_rate': 7.410040941111049e-06, 'epoch': 0.05}\n",
      "{'loss': 0.0, 'grad_norm': 0.0003506238048649344, 'learning_rate': 7.434353962176341e-06, 'epoch': 0.05}\n",
      "{'loss': 0.0228, 'grad_norm': 2.592674951784186, 'learning_rate': 7.458331620518699e-06, 'epoch': 0.05}\n",
      "{'loss': 0.8809, 'grad_norm': 35.500713502402846, 'learning_rate': 7.481983042082595e-06, 'epoch': 0.05}\n",
      "{'loss': 0.0058, 'grad_norm': 0.6666792083593676, 'learning_rate': 7.505316985302266e-06, 'epoch': 0.05}\n",
      "{'loss': 0.0233, 'grad_norm': 1.9817286590085412, 'learning_rate': 7.528341860573942e-06, 'epoch': 0.05}\n",
      "{'loss': 0.0017, 'grad_norm': 0.1893725076202954, 'learning_rate': 7.551065748455211e-06, 'epoch': 0.05}\n",
      "{'loss': 0.5562, 'grad_norm': 18.794788628496033, 'learning_rate': 7.573496416690184e-06, 'epoch': 0.05}\n",
      "{'loss': 0.0447, 'grad_norm': 5.569850394193392, 'learning_rate': 7.595641336150131e-06, 'epoch': 0.05}\n",
      "{'loss': 0.004, 'grad_norm': 0.5663785467278826, 'learning_rate': 7.617507695771499e-06, 'epoch': 0.05}\n",
      "{'loss': 0.0027, 'grad_norm': 0.30380773688268775, 'learning_rate': 7.639102416566009e-06, 'epoch': 0.05}\n",
      "{'loss': 0.0001, 'grad_norm': 0.011763562139437564, 'learning_rate': 7.660432164771186e-06, 'epoch': 0.05}\n",
      "{'loss': 0.0, 'grad_norm': 0.0014761699369449884, 'learning_rate': 7.681503364203827e-06, 'epoch': 0.05}\n",
      "{'loss': 0.0002, 'grad_norm': 0.022216559765147807, 'learning_rate': 7.702322207873675e-06, 'epoch': 0.05}\n",
      "{'loss': 0.0006, 'grad_norm': 0.06359674007945426, 'learning_rate': 7.722894668909854e-06, 'epoch': 0.05}\n",
      "{'loss': 0.0019, 'grad_norm': 0.2518139536122821, 'learning_rate': 7.743226510848278e-06, 'epoch': 0.05}\n",
      "{'loss': 0.0263, 'grad_norm': 2.5267488938122824, 'learning_rate': 7.763323297324384e-06, 'epoch': 0.06}\n",
      "{'loss': 0.0616, 'grad_norm': 7.467135075227276, 'learning_rate': 7.783190401211934e-06, 'epoch': 0.06}\n",
      "{'loss': 0.001, 'grad_norm': 0.1266658025819811, 'learning_rate': 7.802833013245496e-06, 'epoch': 0.06}\n",
      "{'loss': 0.0716, 'grad_norm': 8.741560871755649, 'learning_rate': 7.822256150161167e-06, 'epoch': 0.06}\n",
      "{'loss': 0.005, 'grad_norm': 0.5631213729775463, 'learning_rate': 7.841464662387516e-06, 'epoch': 0.06}\n",
      "{'loss': 0.0178, 'grad_norm': 1.8953219621867126, 'learning_rate': 7.860463241316233e-06, 'epoch': 0.06}\n",
      "{'loss': 0.2737, 'grad_norm': 9.85479717105912, 'learning_rate': 7.879256426179732e-06, 'epoch': 0.06}\n",
      "{'loss': 0.0005, 'grad_norm': 0.05575413276526391, 'learning_rate': 7.897848610560964e-06, 'epoch': 0.06}\n",
      "{'loss': 0.0087, 'grad_norm': 0.9155825504188411, 'learning_rate': 7.916244048558767e-06, 'epoch': 0.06}\n",
      "{'loss': 0.0, 'grad_norm': 0.004205196159683869, 'learning_rate': 7.934446860630398e-06, 'epoch': 0.06}\n",
      "{'loss': 1.0498, 'grad_norm': 32.3598333351172, 'learning_rate': 7.952461039131345e-06, 'epoch': 0.06}\n",
      "{'loss': 0.0001, 'grad_norm': 0.006783047174836668, 'learning_rate': 7.970290453571008e-06, 'epoch': 0.06}\n",
      "{'loss': 0.0035, 'grad_norm': 0.39935320924582485, 'learning_rate': 7.9879388556016e-06, 'epoch': 0.06}\n",
      "{'loss': 0.0001, 'grad_norm': 0.014490093292922888, 'learning_rate': 8.005409883756324e-06, 'epoch': 0.06}\n",
      "{'loss': 0.2354, 'grad_norm': 19.054733144933028, 'learning_rate': 8.02270706795181e-06, 'epoch': 0.06}\n",
      "{'loss': 0.4082, 'grad_norm': 18.139425495980262, 'learning_rate': 8.039833833768753e-06, 'epoch': 0.06}\n",
      "{'loss': 0.1812, 'grad_norm': 13.057449167420321, 'learning_rate': 8.056793506523717e-06, 'epoch': 0.07}\n",
      "{'loss': 0.0, 'grad_norm': 0.00020207495407102672, 'learning_rate': 8.073589315144239e-06, 'epoch': 0.07}\n",
      "{'loss': 0.0, 'grad_norm': 0.000627891635103603, 'learning_rate': 8.0902243958585e-06, 'epoch': 0.07}\n",
      "{'loss': 0.0, 'grad_norm': 0.0009426883088804745, 'learning_rate': 8.106701795710122e-06, 'epoch': 0.07}\n",
      "{'loss': 0.0078, 'grad_norm': 0.897852466674907, 'learning_rate': 8.12302447590796e-06, 'epoch': 0.07}\n",
      "{'loss': 0.0112, 'grad_norm': 0.9509203380448877, 'learning_rate': 8.139195315020065e-06, 'epoch': 0.07}\n",
      "{'loss': 0.0017, 'grad_norm': 0.23133676937278733, 'learning_rate': 8.15521711202045e-06, 'epoch': 0.07}\n",
      "{'loss': 0.4192, 'grad_norm': 25.37948553719741, 'learning_rate': 8.171092589196759e-06, 'epoch': 0.07}\n",
      "{'loss': 0.4204, 'grad_norm': 19.795290238463288, 'learning_rate': 8.186824394926316e-06, 'epoch': 0.07}\n",
      "{'loss': 0.0, 'grad_norm': 0.0014896530194693153, 'learning_rate': 8.20241510632773e-06, 'epoch': 0.07}\n",
      "{'loss': 0.0017, 'grad_norm': 0.2203669561896843, 'learning_rate': 8.21786723179461e-06, 'epoch': 0.07}\n",
      "{'loss': 0.0002, 'grad_norm': 0.020195405795464756, 'learning_rate': 8.233183213417665e-06, 'epoch': 0.07}\n",
      "{'loss': 0.0023, 'grad_norm': 0.2592954611330256, 'learning_rate': 8.248365429301058e-06, 'epoch': 0.07}\n",
      "{'loss': 0.0006, 'grad_norm': 0.05879393197800734, 'learning_rate': 8.26341619577844e-06, 'epoch': 0.07}\n",
      "{'loss': 0.0001, 'grad_norm': 0.01727347406863746, 'learning_rate': 8.278337769533906e-06, 'epoch': 0.07}\n",
      "{'loss': 0.0, 'grad_norm': 0.002755908562058548, 'learning_rate': 8.293132349632674e-06, 'epoch': 0.07}\n",
      "{'loss': 0.0005, 'grad_norm': 0.05451468854281905, 'learning_rate': 8.307802079466086e-06, 'epoch': 0.08}\n",
      "{'loss': 0.002, 'grad_norm': 0.2416422227645724, 'learning_rate': 8.322349048615223e-06, 'epoch': 0.08}\n",
      "{'loss': 0.0001, 'grad_norm': 0.008323108661198885, 'learning_rate': 8.336775294637193e-06, 'epoch': 0.08}\n",
      "{'loss': 0.0015, 'grad_norm': 0.17261882242540805, 'learning_rate': 8.3510828047779e-06, 'epoch': 0.08}\n",
      "{'loss': 0.0, 'grad_norm': 5.458920081315489e-06, 'learning_rate': 8.365273517614908e-06, 'epoch': 0.08}\n",
      "{'loss': 0.0, 'grad_norm': 0.0008269896482066181, 'learning_rate': 8.379349324633788e-06, 'epoch': 0.08}\n",
      "{'loss': 0.0036, 'grad_norm': 0.4163040808751409, 'learning_rate': 8.393312071741149e-06, 'epoch': 0.08}\n",
      "{'loss': 0.0, 'grad_norm': 8.105982145945815e-05, 'learning_rate': 8.407163560717397e-06, 'epoch': 0.08}\n",
      "{'loss': 0.0188, 'grad_norm': 2.5714668997678274, 'learning_rate': 8.420905550612075e-06, 'epoch': 0.08}\n",
      "{'loss': 0.2844, 'grad_norm': 13.91564761026543, 'learning_rate': 8.434539759084453e-06, 'epoch': 0.08}\n",
      "{'loss': 0.0069, 'grad_norm': 0.709971851543088, 'learning_rate': 8.448067863692003e-06, 'epoch': 0.08}\n",
      "{'loss': 0.0, 'grad_norm': 0.00017428519420423528, 'learning_rate': 8.461491503129064e-06, 'epoch': 0.08}\n",
      "{'loss': 0.0, 'grad_norm': 6.997655105811236e-05, 'learning_rate': 8.474812278418107e-06, 'epoch': 0.08}\n",
      "{'loss': 0.001, 'grad_norm': 0.11578711794708124, 'learning_rate': 8.488031754055657e-06, 'epoch': 0.08}\n",
      "{'loss': 0.0001, 'grad_norm': 0.007525409163738824, 'learning_rate': 8.501151459114999e-06, 'epoch': 0.08}\n",
      "{'loss': 0.0003, 'grad_norm': 0.04163791256036061, 'learning_rate': 8.514172888307566e-06, 'epoch': 0.09}\n",
      "{'loss': 0.0001, 'grad_norm': 0.008348799409675504, 'learning_rate': 8.52709750300489e-06, 'epoch': 0.09}\n",
      "{'loss': 0.0543, 'grad_norm': 6.428385383699193, 'learning_rate': 8.53992673222281e-06, 'epoch': 0.09}\n",
      "{'loss': 0.0, 'grad_norm': 0.002303689488268243, 'learning_rate': 8.552661973569658e-06, 'epoch': 0.09}\n",
      "{'loss': 0.0967, 'grad_norm': 5.005368783570887, 'learning_rate': 8.565304594159957e-06, 'epoch': 0.09}\n",
      "{'loss': 0.1658, 'grad_norm': 8.23670859811572, 'learning_rate': 8.577855931495109e-06, 'epoch': 0.09}\n",
      "{'loss': 0.0, 'grad_norm': 0.0012219924190285838, 'learning_rate': 8.590317294312554e-06, 'epoch': 0.09}\n",
      "{'loss': 0.0005, 'grad_norm': 0.05283134428272208, 'learning_rate': 8.602689963404688e-06, 'epoch': 0.09}\n",
      "{'loss': 0.0052, 'grad_norm': 0.5453823154211515, 'learning_rate': 8.614975192408828e-06, 'epoch': 0.09}\n",
      "{'loss': 0.0, 'grad_norm': 0.00016258063092777121, 'learning_rate': 8.627174208569498e-06, 'epoch': 0.09}\n",
      "{'loss': 0.0, 'grad_norm': 0.00016253396472462868, 'learning_rate': 8.639288213474122e-06, 'epoch': 0.09}\n",
      "{'loss': 0.0386, 'grad_norm': 5.388291359684948, 'learning_rate': 8.651318383763265e-06, 'epoch': 0.09}\n",
      "{'loss': 0.0564, 'grad_norm': 6.067112377408025, 'learning_rate': 8.663265871816479e-06, 'epoch': 0.09}\n",
      "{'loss': 0.0004, 'grad_norm': 0.03321977331210523, 'learning_rate': 8.67513180641473e-06, 'epoch': 0.09}\n",
      "{'loss': 0.0052, 'grad_norm': 0.6500985116768186, 'learning_rate': 8.686917293380373e-06, 'epoch': 0.09}\n",
      "{'loss': 0.0, 'grad_norm': 0.0006503855200065899, 'learning_rate': 8.698623416195571e-06, 'epoch': 0.09}\n",
      "{'loss': 0.0031, 'grad_norm': 0.2743100277955953, 'learning_rate': 8.710251236600047e-06, 'epoch': 0.1}\n",
      "{'loss': 0.0003, 'grad_norm': 0.03949286789029727, 'learning_rate': 8.72180179516896e-06, 'epoch': 0.1}\n",
      "{'loss': 0.0002, 'grad_norm': 0.01687772012985608, 'learning_rate': 8.733276111871722e-06, 'epoch': 0.1}\n",
      "{'loss': 0.0, 'grad_norm': 0.0001792015999420221, 'learning_rate': 8.744675186612474e-06, 'epoch': 0.1}\n",
      "{'loss': 0.0, 'grad_norm': 0.00020375864162622634, 'learning_rate': 8.75599999975299e-06, 'epoch': 0.1}\n",
      "{'loss': 0.3359, 'grad_norm': 25.32386053897897, 'learning_rate': 8.767251512618613e-06, 'epoch': 0.1}\n",
      "{'loss': 0.0007, 'grad_norm': 0.08581767886430765, 'learning_rate': 8.778430667987962e-06, 'epoch': 0.1}\n",
      "{'loss': 0.0, 'grad_norm': 0.0016741452476529341, 'learning_rate': 8.789538390566977e-06, 'epoch': 0.1}\n",
      "{'loss': 0.0, 'grad_norm': 0.003966235975769844, 'learning_rate': 8.800575587447912e-06, 'epoch': 0.1}\n",
      "{'loss': 0.0, 'grad_norm': 0.0003621518179776956, 'learning_rate': 8.811543148553846e-06, 'epoch': 0.1}\n",
      "{'loss': 0.0684, 'grad_norm': 11.08933258222246, 'learning_rate': 8.822441947069277e-06, 'epoch': 0.1}\n",
      "{'loss': 0.0, 'grad_norm': 0.0005651242073255558, 'learning_rate': 8.833272839857288e-06, 'epoch': 0.1}\n",
      "{'loss': 0.0026, 'grad_norm': 0.2690413625568831, 'learning_rate': 8.844036667863787e-06, 'epoch': 0.1}\n",
      "{'loss': 0.0001, 'grad_norm': 0.005948973298077562, 'learning_rate': 8.854734256509333e-06, 'epoch': 0.1}\n",
      "{'loss': 0.0369, 'grad_norm': 4.115289297583782, 'learning_rate': 8.865366416068965e-06, 'epoch': 0.1}\n",
      "{'loss': 0.0018, 'grad_norm': 0.2163519718944715, 'learning_rate': 8.87593394204048e-06, 'epoch': 0.1}\n",
      "{'loss': 0.0, 'grad_norm': 0.00014395157852293135, 'learning_rate': 8.886437615501607e-06, 'epoch': 0.11}\n",
      "{'loss': 0.0, 'grad_norm': 0.00038402104663420915, 'learning_rate': 8.896878203456422e-06, 'epoch': 0.11}\n",
      "{'loss': 0.0048, 'grad_norm': 0.5485321278460471, 'learning_rate': 8.907256459171455e-06, 'epoch': 0.11}\n",
      "{'loss': 0.0, 'grad_norm': 0.00023456817084758388, 'learning_rate': 8.917573122501803e-06, 'epoch': 0.11}\n",
      "{'loss': 0.0, 'grad_norm': 0.00020892369232796112, 'learning_rate': 8.927828920207633e-06, 'epoch': 0.11}\n",
      "{'loss': 0.0037, 'grad_norm': 0.4292648664487294, 'learning_rate': 8.938024566261389e-06, 'epoch': 0.11}\n",
      "{'loss': 0.0006, 'grad_norm': 0.18795061601097413, 'learning_rate': 8.948160762146057e-06, 'epoch': 0.11}\n",
      "{'loss': 0.0089, 'grad_norm': 1.0757080296993264, 'learning_rate': 8.958238197144793e-06, 'epoch': 0.11}\n",
      "{'loss': 0.0001, 'grad_norm': 0.010423151930465545, 'learning_rate': 8.968257548622162e-06, 'epoch': 0.11}\n",
      "{'loss': 0.0, 'grad_norm': 9.268455183148281e-05, 'learning_rate': 8.97821948229738e-06, 'epoch': 0.11}\n",
      "{'loss': 0.0001, 'grad_norm': 0.011913184740415278, 'learning_rate': 8.988124652509712e-06, 'epoch': 0.11}\n",
      "{'loss': 0.2135, 'grad_norm': 18.943584811686993, 'learning_rate': 8.997973702476397e-06, 'epoch': 0.11}\n",
      "{'loss': 0.0025, 'grad_norm': 0.3156272812562592, 'learning_rate': 9.007767264543275e-06, 'epoch': 0.11}\n",
      "{'loss': 0.0002, 'grad_norm': 0.028135867511035473, 'learning_rate': 9.017505960428414e-06, 'epoch': 0.11}\n",
      "{'loss': 0.0, 'grad_norm': 0.004130872418955687, 'learning_rate': 9.027190401458944e-06, 'epoch': 0.11}\n",
      "{'loss': 0.051, 'grad_norm': 4.20758682025405, 'learning_rate': 9.036821188801332e-06, 'epoch': 0.11}\n",
      "{'loss': 0.1466, 'grad_norm': 13.462766375868332, 'learning_rate': 9.046398913685295e-06, 'epoch': 0.12}\n",
      "{'loss': 0.0, 'grad_norm': 6.778500392635904e-05, 'learning_rate': 9.055924157621623e-06, 'epoch': 0.12}\n",
      "{'loss': 0.0, 'grad_norm': 0.0007853603536167136, 'learning_rate': 9.065397492614013e-06, 'epoch': 0.12}\n",
      "{'loss': 0.0001, 'grad_norm': 0.006440859623730699, 'learning_rate': 9.074819481365197e-06, 'epoch': 0.12}\n",
      "{'loss': 0.0003, 'grad_norm': 0.029244644575337903, 'learning_rate': 9.084190677477512e-06, 'epoch': 0.12}\n",
      "{'loss': 0.0003, 'grad_norm': 0.03736052727806903, 'learning_rate': 9.093511625648068e-06, 'epoch': 0.12}\n",
      "{'loss': 0.0001, 'grad_norm': 0.010174631099650824, 'learning_rate': 9.102782861858744e-06, 'epoch': 0.12}\n",
      "{'loss': 0.0008, 'grad_norm': 0.07430654904012342, 'learning_rate': 9.112004913561122e-06, 'epoch': 0.12}\n",
      "{'loss': 0.0027, 'grad_norm': 0.32579332644905806, 'learning_rate': 9.121178299856546e-06, 'epoch': 0.12}\n",
      "{'loss': 0.0002, 'grad_norm': 0.014505816596859124, 'learning_rate': 9.130303531671462e-06, 'epoch': 0.12}\n",
      "{'loss': 0.0, 'grad_norm': 0.0002476820649320146, 'learning_rate': 9.139381111928178e-06, 'epoch': 0.12}\n",
      "{'loss': 1.7227, 'grad_norm': 27.66979938220663, 'learning_rate': 9.148411535711168e-06, 'epoch': 0.12}\n",
      "{'loss': 0.0021, 'grad_norm': 0.2935194688534642, 'learning_rate': 9.157395290429125e-06, 'epoch': 0.12}\n",
      "{'loss': 0.0, 'grad_norm': 0.0015388907769408647, 'learning_rate': 9.166332855972787e-06, 'epoch': 0.12}\n",
      "{'loss': 0.0011, 'grad_norm': 0.09953779098031554, 'learning_rate': 9.175224704868788e-06, 'epoch': 0.12}\n",
      "{'loss': 0.0001, 'grad_norm': 0.006615538030624796, 'learning_rate': 9.184071302429544e-06, 'epoch': 0.13}\n",
      "{'loss': 0.0013, 'grad_norm': 0.15272381641155983, 'learning_rate': 9.192873106899379e-06, 'epoch': 0.13}\n",
      "{'loss': 0.0001, 'grad_norm': 0.007146739373684103, 'learning_rate': 9.201630569596949e-06, 'epoch': 0.13}\n",
      "{'loss': 0.0, 'grad_norm': 0.00042865926714917, 'learning_rate': 9.210344135054102e-06, 'epoch': 0.13}\n",
      "{'loss': 0.7959, 'grad_norm': 28.307491710041397, 'learning_rate': 9.219014241151289e-06, 'epoch': 0.13}\n",
      "{'loss': 0.0002, 'grad_norm': 0.023118996618198943, 'learning_rate': 9.22764131924959e-06, 'epoch': 0.13}\n",
      "{'loss': 0.0, 'grad_norm': 0.002768596860881457, 'learning_rate': 9.236225794319495e-06, 'epoch': 0.13}\n",
      "{'loss': 0.0663, 'grad_norm': 9.214921162641133, 'learning_rate': 9.24476808506653e-06, 'epoch': 0.13}\n",
      "{'loss': 0.0001, 'grad_norm': 0.015568199703544882, 'learning_rate': 9.25326860405379e-06, 'epoch': 0.13}\n",
      "{'loss': 0.0014, 'grad_norm': 0.11718327594137819, 'learning_rate': 9.261727757821498e-06, 'epoch': 0.13}\n",
      "{'loss': 0.0005, 'grad_norm': 0.05390961056061274, 'learning_rate': 9.270145947003678e-06, 'epoch': 0.13}\n",
      "{'loss': 0.191, 'grad_norm': 14.25578859779149, 'learning_rate': 9.278523566442019e-06, 'epoch': 0.13}\n",
      "{'loss': 0.0, 'grad_norm': 0.0006747447767368555, 'learning_rate': 9.28686100529698e-06, 'epoch': 0.13}\n",
      "{'loss': 0.437, 'grad_norm': 20.138658899623646, 'learning_rate': 9.295158647156278e-06, 'epoch': 0.13}\n",
      "{'loss': 0.0005, 'grad_norm': 0.05138136156866065, 'learning_rate': 9.303416870140782e-06, 'epoch': 0.13}\n",
      "{'loss': 0.001, 'grad_norm': 0.12250046282311026, 'learning_rate': 9.311636047007902e-06, 'epoch': 0.13}\n",
      "{'loss': 0.0001, 'grad_norm': 0.009403121591696251, 'learning_rate': 9.31981654525255e-06, 'epoch': 0.14}\n",
      "{'loss': 0.1379, 'grad_norm': 8.266141878923927, 'learning_rate': 9.32795872720574e-06, 'epoch': 0.14}\n",
      "{'loss': 0.0019, 'grad_norm': 0.1866309370462203, 'learning_rate': 9.336062950130882e-06, 'epoch': 0.14}\n",
      "{'loss': 0.0009, 'grad_norm': 0.09814285637484466, 'learning_rate': 9.344129566317845e-06, 'epoch': 0.14}\n",
      "{'loss': 0.0, 'grad_norm': 0.0012859570592625442, 'learning_rate': 9.352158923174845e-06, 'epoch': 0.14}\n",
      "{'loss': 0.0111, 'grad_norm': 1.2383528727807813, 'learning_rate': 9.36015136331823e-06, 'epoch': 0.14}\n",
      "{'loss': 0.0, 'grad_norm': 0.001884729616154413, 'learning_rate': 9.368107224660202e-06, 'epoch': 0.14}\n",
      "{'loss': 0.0, 'grad_norm': 0.00563408961612702, 'learning_rate': 9.376026840494538e-06, 'epoch': 0.14}\n",
      "{'loss': 0.0, 'grad_norm': 0.0002280744686942088, 'learning_rate': 9.383910539580373e-06, 'epoch': 0.14}\n",
      "{'loss': 0.0003, 'grad_norm': 0.026934361535056347, 'learning_rate': 9.391758646224096e-06, 'epoch': 0.14}\n",
      "{'loss': 0.0081, 'grad_norm': 0.7973945644243153, 'learning_rate': 9.399571480359392e-06, 'epoch': 0.14}\n",
      "{'loss': 0.0001, 'grad_norm': 0.009249397666215697, 'learning_rate': 9.407349357625511e-06, 'epoch': 0.14}\n",
      "{'loss': 0.0, 'grad_norm': 0.0009020462885609743, 'learning_rate': 9.415092589443769e-06, 'epoch': 0.14}\n",
      "{'loss': 0.0008, 'grad_norm': 0.10078347355215689, 'learning_rate': 9.422801483092389e-06, 'epoch': 0.14}\n",
      "{'loss': 0.0, 'grad_norm': 0.00014790128457654625, 'learning_rate': 9.430476341779646e-06, 'epoch': 0.14}\n",
      "{'loss': 0.0011, 'grad_norm': 0.11210813322276719, 'learning_rate': 9.438117464715445e-06, 'epoch': 0.14}\n",
      "{'loss': 0.0133, 'grad_norm': 1.5148723345809492, 'learning_rate': 9.445725147181306e-06, 'epoch': 0.15}\n",
      "{'loss': 0.0189, 'grad_norm': 2.378249906249793, 'learning_rate': 9.453299680598836e-06, 'epoch': 0.15}\n",
      "{'loss': 0.0001, 'grad_norm': 0.008839346669581537, 'learning_rate': 9.460841352596712e-06, 'epoch': 0.15}\n",
      "{'loss': 0.025, 'grad_norm': 2.797509659519513, 'learning_rate': 9.46835044707622e-06, 'epoch': 0.15}\n",
      "{'loss': 0.0, 'grad_norm': 0.005627574526025301, 'learning_rate': 9.475827244275373e-06, 'epoch': 0.15}\n",
      "{'loss': 0.0, 'grad_norm': 0.000418120087289687, 'learning_rate': 9.483272020831686e-06, 'epoch': 0.15}\n",
      "{'loss': 0.0, 'grad_norm': 0.0016457079673598822, 'learning_rate': 9.490685049843568e-06, 'epoch': 0.15}\n",
      "{'loss': 0.004, 'grad_norm': 0.5606810512186752, 'learning_rate': 9.498066600930455e-06, 'epoch': 0.15}\n",
      "{'loss': 0.0003, 'grad_norm': 0.0386065064276979, 'learning_rate': 9.505416940291635e-06, 'epoch': 0.15}\n",
      "{'loss': 0.0025, 'grad_norm': 0.25221493152293883, 'learning_rate': 9.512736330763865e-06, 'epoch': 0.15}\n",
      "{'loss': 0.0004, 'grad_norm': 0.040724601744872954, 'learning_rate': 9.520025031877758e-06, 'epoch': 0.15}\n",
      "{'loss': 0.0002, 'grad_norm': 0.02385219502469358, 'learning_rate': 9.527283299913002e-06, 'epoch': 0.15}\n",
      "{'loss': 0.0122, 'grad_norm': 1.2483556422203974, 'learning_rate': 9.53451138795243e-06, 'epoch': 0.15}\n",
      "{'loss': 0.0001, 'grad_norm': 0.005095174592034607, 'learning_rate': 9.541709545934973e-06, 'epoch': 0.15}\n",
      "{'loss': 0.001, 'grad_norm': 0.1342821987314779, 'learning_rate': 9.54887802070751e-06, 'epoch': 0.15}\n",
      "{'loss': 0.0, 'grad_norm': 2.1204408663901515e-05, 'learning_rate': 9.55601705607568e-06, 'epoch': 0.15}\n",
      "{'loss': 0.0001, 'grad_norm': 0.010026294810654453, 'learning_rate': 9.563126892853612e-06, 'epoch': 0.16}\n",
      "{'loss': 0.0, 'grad_norm': 0.00014014875374689125, 'learning_rate': 9.570207768912687e-06, 'epoch': 0.16}\n",
      "{'loss': 0.0005, 'grad_norm': 0.06203841776057057, 'learning_rate': 9.577259919229286e-06, 'epoch': 0.16}\n",
      "{'loss': 0.0, 'grad_norm': 0.001175108944249278, 'learning_rate': 9.584283575931568e-06, 'epoch': 0.16}\n",
      "{'loss': 0.0046, 'grad_norm': 0.5165143002398082, 'learning_rate': 9.591278968345328e-06, 'epoch': 0.16}\n",
      "{'loss': 0.0, 'grad_norm': 0.005240807854468919, 'learning_rate': 9.598246323038927e-06, 'epoch': 0.16}\n",
      "{'loss': 1.4277, 'grad_norm': 33.07771352188566, 'learning_rate': 9.60518586386731e-06, 'epoch': 0.16}\n",
      "{'loss': 0.0002, 'grad_norm': 0.025152485788717115, 'learning_rate': 9.612097812015178e-06, 'epoch': 0.16}\n",
      "{'loss': 0.1694, 'grad_norm': 9.62688080877941, 'learning_rate': 9.61898238603927e-06, 'epoch': 0.16}\n",
      "{'loss': 0.0024, 'grad_norm': 0.2888895883701998, 'learning_rate': 9.625839801909852e-06, 'epoch': 0.16}\n",
      "{'loss': 0.0005, 'grad_norm': 0.03882763559840843, 'learning_rate': 9.632670273051355e-06, 'epoch': 0.16}\n",
      "{'loss': 0.1205, 'grad_norm': 9.270393366857176, 'learning_rate': 9.639474010382234e-06, 'epoch': 0.16}\n",
      "{'loss': 0.0679, 'grad_norm': 7.3906985515136325, 'learning_rate': 9.646251222354047e-06, 'epoch': 0.16}\n",
      "{'loss': 0.0001, 'grad_norm': 0.008636354556782567, 'learning_rate': 9.65300211498978e-06, 'epoch': 0.16}\n",
      "{'loss': 0.0031, 'grad_norm': 0.3544678692844161, 'learning_rate': 9.659726891921429e-06, 'epoch': 0.16}\n",
      "{'loss': 0.7959, 'grad_norm': 22.580513227929934, 'learning_rate': 9.666425754426843e-06, 'epoch': 0.17}\n",
      "{'loss': 0.0021, 'grad_norm': 0.21511501180235365, 'learning_rate': 9.673098901465887e-06, 'epoch': 0.17}\n",
      "{'loss': 0.1976, 'grad_norm': 18.996921490679664, 'learning_rate': 9.679746529715885e-06, 'epoch': 0.17}\n",
      "{'loss': 0.0004, 'grad_norm': 0.06095668325719044, 'learning_rate': 9.686368833606421e-06, 'epoch': 0.17}\n",
      "{'loss': 0.0001, 'grad_norm': 0.010365802671615217, 'learning_rate': 9.692966005353435e-06, 'epoch': 0.17}\n",
      "{'loss': 1.5166, 'grad_norm': 25.048251775167614, 'learning_rate': 9.699538234992726e-06, 'epoch': 0.17}\n",
      "{'loss': 0.1159, 'grad_norm': 10.912564348902842, 'learning_rate': 9.706085710412776e-06, 'epoch': 0.17}\n",
      "{'loss': 0.0, 'grad_norm': 0.0011928100336209804, 'learning_rate': 9.712608617386999e-06, 'epoch': 0.17}\n",
      "{'loss': 0.0148, 'grad_norm': 1.56246528586926, 'learning_rate': 9.719107139605345e-06, 'epoch': 0.17}\n",
      "{'loss': 0.0, 'grad_norm': 0.0011473754349389821, 'learning_rate': 9.72558145870537e-06, 'epoch': 0.17}\n",
      "{'loss': 0.0401, 'grad_norm': 3.838044933682234, 'learning_rate': 9.73203175430267e-06, 'epoch': 0.17}\n",
      "{'loss': 0.0, 'grad_norm': 0.0007402956389527396, 'learning_rate': 9.738458204020791e-06, 'epoch': 0.17}\n",
      "{'loss': 0.0015, 'grad_norm': 0.18049037935845336, 'learning_rate': 9.744860983520588e-06, 'epoch': 0.17}\n",
      "{'loss': 0.4207, 'grad_norm': 25.53436118398537, 'learning_rate': 9.751240266529019e-06, 'epoch': 0.17}\n",
      "{'loss': 0.0001, 'grad_norm': 0.003966864233428802, 'learning_rate': 9.757596224867439e-06, 'epoch': 0.17}\n",
      "{'loss': 0.0835, 'grad_norm': 9.320264613115665, 'learning_rate': 9.763929028479363e-06, 'epoch': 0.17}\n",
      "{'loss': 0.0, 'grad_norm': 0.0006016525557005614, 'learning_rate': 9.770238845457734e-06, 'epoch': 0.18}\n",
      "{'loss': 0.0845, 'grad_norm': 8.818735067786127, 'learning_rate': 9.7765258420717e-06, 'epoch': 0.18}\n",
      "{'loss': 0.4756, 'grad_norm': 27.735506138905237, 'learning_rate': 9.782790182792888e-06, 'epoch': 0.18}\n",
      "{'loss': 0.0, 'grad_norm': 0.0022954318739826863, 'learning_rate': 9.789032030321233e-06, 'epoch': 0.18}\n",
      "{'loss': 0.0035, 'grad_norm': 0.3657901839086353, 'learning_rate': 9.795251545610334e-06, 'epoch': 0.18}\n",
      "{'loss': 0.0001, 'grad_norm': 0.0070433984490409505, 'learning_rate': 9.80144888789235e-06, 'epoch': 0.18}\n",
      "{'loss': 0.0001, 'grad_norm': 0.006615471215702845, 'learning_rate': 9.807624214702467e-06, 'epoch': 0.18}\n",
      "{'loss': 0.001, 'grad_norm': 0.12234600596704745, 'learning_rate': 9.81377768190292e-06, 'epoch': 0.18}\n",
      "{'loss': 0.0189, 'grad_norm': 2.054614743877318, 'learning_rate': 9.819909443706607e-06, 'epoch': 0.18}\n",
      "{'loss': 0.0, 'grad_norm': 5.51570079555793e-05, 'learning_rate': 9.82601965270027e-06, 'epoch': 0.18}\n",
      "{'loss': 0.0918, 'grad_norm': 9.419740653385887, 'learning_rate': 9.832108459867276e-06, 'epoch': 0.18}\n",
      "{'loss': 0.0003, 'grad_norm': 0.03475118776648241, 'learning_rate': 9.838176014610022e-06, 'epoch': 0.18}\n",
      "{'loss': 0.0045, 'grad_norm': 0.5468596865008593, 'learning_rate': 9.844222464771901e-06, 'epoch': 0.18}\n",
      "{'loss': 0.0003, 'grad_norm': 0.03372121266439379, 'learning_rate': 9.850247956658943e-06, 'epoch': 0.18}\n",
      "{'loss': 0.0023, 'grad_norm': 0.191144374838383, 'learning_rate': 9.856252635061042e-06, 'epoch': 0.18}\n",
      "{'loss': 0.0724, 'grad_norm': 9.357495458317969, 'learning_rate': 9.862236643272848e-06, 'epoch': 0.18}\n",
      "{'loss': 0.0, 'grad_norm': 0.0031265932846622517, 'learning_rate': 9.868200123114258e-06, 'epoch': 0.19}\n",
      "{'loss': 0.0575, 'grad_norm': 6.314147130016229, 'learning_rate': 9.874143214950616e-06, 'epoch': 0.19}\n",
      "{'loss': 0.0, 'grad_norm': 0.0014069830011972768, 'learning_rate': 9.88006605771251e-06, 'epoch': 0.19}\n",
      "{'loss': 0.0001, 'grad_norm': 0.0053646914094565845, 'learning_rate': 9.885968788915278e-06, 'epoch': 0.19}\n",
      "{'loss': 0.1049, 'grad_norm': 7.426105907553118, 'learning_rate': 9.891851544678152e-06, 'epoch': 0.19}\n",
      "{'loss': 0.0201, 'grad_norm': 1.9082549760659397, 'learning_rate': 9.897714459743102e-06, 'epoch': 0.19}\n",
      "{'loss': 0.0127, 'grad_norm': 1.072740071154954, 'learning_rate': 9.90355766749335e-06, 'epoch': 0.19}\n",
      "{'loss': 0.0, 'grad_norm': 0.0017292207474545061, 'learning_rate': 9.909381299971577e-06, 'epoch': 0.19}\n",
      "{'loss': 0.0001, 'grad_norm': 0.009376919678379303, 'learning_rate': 9.915185487897825e-06, 'epoch': 0.19}\n",
      "{'loss': 0.9346, 'grad_norm': 27.96152905213865, 'learning_rate': 9.920970360687114e-06, 'epoch': 0.19}\n",
      "{'loss': 0.001, 'grad_norm': 0.10010740186188523, 'learning_rate': 9.92673604646674e-06, 'epoch': 0.19}\n",
      "{'loss': 0.0153, 'grad_norm': 1.6178175578812926, 'learning_rate': 9.932482672093313e-06, 'epoch': 0.19}\n",
      "{'loss': 0.0176, 'grad_norm': 1.6049877476447356, 'learning_rate': 9.938210363169501e-06, 'epoch': 0.19}\n",
      "{'loss': 0.0143, 'grad_norm': 1.2441097716813094, 'learning_rate': 9.943919244060504e-06, 'epoch': 0.19}\n",
      "{'loss': 0.0108, 'grad_norm': 0.9010188627205542, 'learning_rate': 9.949609437910255e-06, 'epoch': 0.19}\n",
      "{'loss': 0.0176, 'grad_norm': 1.7578768570033048, 'learning_rate': 9.955281066657357e-06, 'epoch': 0.19}\n",
      "{'loss': 0.0001, 'grad_norm': 0.004165575398846052, 'learning_rate': 9.960934251050768e-06, 'epoch': 0.2}\n",
      "{'loss': 0.0895, 'grad_norm': 7.857031774354909, 'learning_rate': 9.96656911066522e-06, 'epoch': 0.2}\n",
      "{'loss': 0.0136, 'grad_norm': 1.6409313824085001, 'learning_rate': 9.972185763916392e-06, 'epoch': 0.2}\n",
      "{'loss': 0.0001, 'grad_norm': 0.012968544040080023, 'learning_rate': 9.977784328075851e-06, 'epoch': 0.2}\n",
      "{'loss': 0.0003, 'grad_norm': 0.039256669794124545, 'learning_rate': 9.983364919285742e-06, 'epoch': 0.2}\n",
      "{'loss': 0.0001, 'grad_norm': 0.012217827433481364, 'learning_rate': 9.988927652573233e-06, 'epoch': 0.2}\n",
      "{'loss': 0.0001, 'grad_norm': 0.005600997704195384, 'learning_rate': 9.994472641864756e-06, 'epoch': 0.2}\n",
      "{'loss': 0.0002, 'grad_norm': 0.019069991745296572, 'learning_rate': 1e-05, 'epoch': 0.2}\n",
      "{'loss': 0.1221, 'grad_norm': 10.511170394583319, 'learning_rate': 1e-05, 'epoch': 0.2}\n",
      "{'loss': 0.1941, 'grad_norm': 11.918708110669703, 'learning_rate': 9.99647266313933e-06, 'epoch': 0.2}\n",
      "{'loss': 0.0267, 'grad_norm': 3.5891821981223466, 'learning_rate': 9.99294532627866e-06, 'epoch': 0.2}\n",
      "{'loss': 0.0003, 'grad_norm': 0.036275342505005186, 'learning_rate': 9.989417989417989e-06, 'epoch': 0.2}\n",
      "{'loss': 0.8838, 'grad_norm': 27.914636460304074, 'learning_rate': 9.98589065255732e-06, 'epoch': 0.2}\n",
      "{'loss': 0.623, 'grad_norm': 26.759531194802815, 'learning_rate': 9.982363315696649e-06, 'epoch': 0.2}\n",
      "{'loss': 0.0003, 'grad_norm': 0.036973476972604426, 'learning_rate': 9.97883597883598e-06, 'epoch': 0.2}\n",
      "{'loss': 0.0, 'grad_norm': 0.000500899448008299, 'learning_rate': 9.97530864197531e-06, 'epoch': 0.21}\n",
      "{'loss': 0.0622, 'grad_norm': 7.085207463082636, 'learning_rate': 9.97178130511464e-06, 'epoch': 0.21}\n",
      "{'loss': 0.0002, 'grad_norm': 0.027266435652106798, 'learning_rate': 9.968253968253969e-06, 'epoch': 0.21}\n",
      "{'loss': 0.0007, 'grad_norm': 0.08415959897391309, 'learning_rate': 9.9647266313933e-06, 'epoch': 0.21}\n",
      "{'loss': 0.0003, 'grad_norm': 0.03100230166531855, 'learning_rate': 9.961199294532629e-06, 'epoch': 0.21}\n",
      "{'loss': 0.0001, 'grad_norm': 0.005825326259385087, 'learning_rate': 9.957671957671959e-06, 'epoch': 0.21}\n",
      "{'loss': 0.0, 'grad_norm': 0.0028786165970295873, 'learning_rate': 9.954144620811288e-06, 'epoch': 0.21}\n",
      "{'loss': 0.0, 'grad_norm': 0.0006423861991158987, 'learning_rate': 9.950617283950618e-06, 'epoch': 0.21}\n",
      "{'loss': 0.0, 'grad_norm': 0.0013864048369873576, 'learning_rate': 9.947089947089947e-06, 'epoch': 0.21}\n",
      "{'loss': 0.0002, 'grad_norm': 0.014644311780873645, 'learning_rate': 9.943562610229278e-06, 'epoch': 0.21}\n",
      "{'loss': 0.3237, 'grad_norm': 28.076907666988987, 'learning_rate': 9.940035273368608e-06, 'epoch': 0.21}\n",
      "{'loss': 0.0001, 'grad_norm': 0.005841845190652072, 'learning_rate': 9.936507936507937e-06, 'epoch': 0.21}\n",
      "{'loss': 0.0011, 'grad_norm': 0.09461627025504547, 'learning_rate': 9.932980599647268e-06, 'epoch': 0.21}\n",
      "{'loss': 0.0657, 'grad_norm': 6.705948810622073, 'learning_rate': 9.929453262786597e-06, 'epoch': 0.21}\n",
      "{'loss': 0.0372, 'grad_norm': 3.8697813024506162, 'learning_rate': 9.925925925925927e-06, 'epoch': 0.21}\n",
      "{'loss': 0.0087, 'grad_norm': 0.8870920076270147, 'learning_rate': 9.922398589065256e-06, 'epoch': 0.21}\n",
      "{'loss': 0.0, 'grad_norm': 0.003204100037421878, 'learning_rate': 9.918871252204587e-06, 'epoch': 0.22}\n",
      "{'loss': 0.0, 'grad_norm': 0.002433242970985921, 'learning_rate': 9.915343915343916e-06, 'epoch': 0.22}\n",
      "{'loss': 0.0406, 'grad_norm': 4.242614836973447, 'learning_rate': 9.911816578483246e-06, 'epoch': 0.22}\n",
      "{'loss': 0.4226, 'grad_norm': 22.740253750233308, 'learning_rate': 9.908289241622577e-06, 'epoch': 0.22}\n",
      "{'loss': 0.0007, 'grad_norm': 0.07440215275793476, 'learning_rate': 9.904761904761906e-06, 'epoch': 0.22}\n",
      "{'loss': 0.0, 'grad_norm': 5.554640642032555e-05, 'learning_rate': 9.901234567901236e-06, 'epoch': 0.22}\n",
      "{'loss': 0.067, 'grad_norm': 8.687311184155403, 'learning_rate': 9.897707231040565e-06, 'epoch': 0.22}\n",
      "{'loss': 0.0001, 'grad_norm': 0.017548220250582395, 'learning_rate': 9.894179894179896e-06, 'epoch': 0.22}\n",
      "{'loss': 0.0172, 'grad_norm': 1.9011001438570423, 'learning_rate': 9.890652557319224e-06, 'epoch': 0.22}\n",
      "{'loss': 0.0, 'grad_norm': 0.002164573169764195, 'learning_rate': 9.887125220458555e-06, 'epoch': 0.22}\n",
      "{'loss': 0.0, 'grad_norm': 0.00025659299058053376, 'learning_rate': 9.883597883597884e-06, 'epoch': 0.22}\n",
      "{'loss': 0.0002, 'grad_norm': 0.02525347798907043, 'learning_rate': 9.880070546737214e-06, 'epoch': 0.22}\n",
      "{'loss': 0.0, 'grad_norm': 0.002894717800956254, 'learning_rate': 9.876543209876543e-06, 'epoch': 0.22}\n",
      "{'loss': 0.0, 'grad_norm': 5.971994275383773e-05, 'learning_rate': 9.873015873015874e-06, 'epoch': 0.22}\n",
      "{'loss': 0.0052, 'grad_norm': 0.518502902733707, 'learning_rate': 9.869488536155204e-06, 'epoch': 0.22}\n",
      "{'loss': 0.0004, 'grad_norm': 0.0311438865527313, 'learning_rate': 9.865961199294533e-06, 'epoch': 0.22}\n",
      "{'loss': 0.0, 'grad_norm': 0.00032698138963533984, 'learning_rate': 9.862433862433864e-06, 'epoch': 0.23}\n",
      "{'loss': 0.0, 'grad_norm': 0.00018692428089558545, 'learning_rate': 9.858906525573193e-06, 'epoch': 0.23}\n",
      "{'loss': 0.0002, 'grad_norm': 0.019011633719173197, 'learning_rate': 9.855379188712523e-06, 'epoch': 0.23}\n",
      "{'loss': 0.0014, 'grad_norm': 0.13896780942431722, 'learning_rate': 9.851851851851852e-06, 'epoch': 0.23}\n",
      "{'loss': 0.0023, 'grad_norm': 0.30673422872980854, 'learning_rate': 9.848324514991183e-06, 'epoch': 0.23}\n",
      "{'loss': 0.0479, 'grad_norm': 4.061192933136905, 'learning_rate': 9.844797178130512e-06, 'epoch': 0.23}\n",
      "{'loss': 0.0, 'grad_norm': 0.0003627841294685361, 'learning_rate': 9.841269841269842e-06, 'epoch': 0.23}\n",
      "{'loss': 0.0, 'grad_norm': 8.961601454647273e-05, 'learning_rate': 9.837742504409173e-06, 'epoch': 0.23}\n",
      "{'loss': 0.0501, 'grad_norm': 6.51282556346079, 'learning_rate': 9.834215167548502e-06, 'epoch': 0.23}\n",
      "{'loss': 0.0008, 'grad_norm': 0.10736230585724935, 'learning_rate': 9.830687830687832e-06, 'epoch': 0.23}\n",
      "{'loss': 0.0035, 'grad_norm': 0.39889175101802704, 'learning_rate': 9.827160493827161e-06, 'epoch': 0.23}\n",
      "{'loss': 0.0012, 'grad_norm': 0.153110889592384, 'learning_rate': 9.823633156966492e-06, 'epoch': 0.23}\n",
      "{'loss': 0.0006, 'grad_norm': 0.06444548070254176, 'learning_rate': 9.82010582010582e-06, 'epoch': 0.23}\n",
      "{'loss': 0.0853, 'grad_norm': 8.486958540666738, 'learning_rate': 9.816578483245151e-06, 'epoch': 0.23}\n",
      "{'loss': 0.1031, 'grad_norm': 9.011390683370678, 'learning_rate': 9.81305114638448e-06, 'epoch': 0.23}\n",
      "{'loss': 0.005, 'grad_norm': 0.6946434273684176, 'learning_rate': 9.80952380952381e-06, 'epoch': 0.23}\n",
      "{'loss': 0.0175, 'grad_norm': 1.745648150155651, 'learning_rate': 9.80599647266314e-06, 'epoch': 0.24}\n",
      "{'loss': 0.0055, 'grad_norm': 0.5786281149206172, 'learning_rate': 9.80246913580247e-06, 'epoch': 0.24}\n",
      "{'loss': 0.0, 'grad_norm': 0.0005490241001501566, 'learning_rate': 9.7989417989418e-06, 'epoch': 0.24}\n",
      "{'loss': 0.1185, 'grad_norm': 7.0890508875710765, 'learning_rate': 9.79541446208113e-06, 'epoch': 0.24}\n",
      "{'loss': 0.0, 'grad_norm': 0.004133020500708654, 'learning_rate': 9.79188712522046e-06, 'epoch': 0.24}\n",
      "{'loss': 0.0001, 'grad_norm': 0.004401366188583172, 'learning_rate': 9.788359788359789e-06, 'epoch': 0.24}\n",
      "{'loss': 0.017, 'grad_norm': 1.3898525450043842, 'learning_rate': 9.78483245149912e-06, 'epoch': 0.24}\n",
      "{'loss': 0.0052, 'grad_norm': 0.4415710782880179, 'learning_rate': 9.781305114638448e-06, 'epoch': 0.24}\n",
      "{'loss': 0.0, 'grad_norm': 7.677172246444142e-05, 'learning_rate': 9.777777777777779e-06, 'epoch': 0.24}\n",
      "{'loss': 0.0558, 'grad_norm': 4.350587362338431, 'learning_rate': 9.774250440917108e-06, 'epoch': 0.24}\n",
      "{'loss': 0.0002, 'grad_norm': 0.0200720931498879, 'learning_rate': 9.770723104056438e-06, 'epoch': 0.24}\n",
      "{'loss': 0.0, 'grad_norm': 0.002285988071029741, 'learning_rate': 9.767195767195769e-06, 'epoch': 0.24}\n",
      "{'loss': 0.0, 'grad_norm': 0.0007031716949633706, 'learning_rate': 9.763668430335098e-06, 'epoch': 0.24}\n",
      "{'loss': 0.048, 'grad_norm': 4.867100698024638, 'learning_rate': 9.760141093474428e-06, 'epoch': 0.24}\n",
      "{'loss': 0.0003, 'grad_norm': 0.030848961254567937, 'learning_rate': 9.756613756613757e-06, 'epoch': 0.24}\n",
      "{'loss': 0.0514, 'grad_norm': 6.409224549996309, 'learning_rate': 9.753086419753087e-06, 'epoch': 0.25}\n",
      "{'loss': 0.3269, 'grad_norm': 18.079866185326065, 'learning_rate': 9.749559082892416e-06, 'epoch': 0.25}\n",
      "{'loss': 0.0227, 'grad_norm': 2.478086465966512, 'learning_rate': 9.746031746031747e-06, 'epoch': 0.25}\n",
      "{'loss': 0.0004, 'grad_norm': 0.05641561035240476, 'learning_rate': 9.742504409171076e-06, 'epoch': 0.25}\n",
      "{'loss': 0.2073, 'grad_norm': 12.571428001701046, 'learning_rate': 9.738977072310406e-06, 'epoch': 0.25}\n",
      "{'loss': 0.0004, 'grad_norm': 0.053259703227685105, 'learning_rate': 9.735449735449735e-06, 'epoch': 0.25}\n",
      "{'loss': 0.0, 'grad_norm': 0.00015299464494716904, 'learning_rate': 9.731922398589066e-06, 'epoch': 0.25}\n",
      "{'loss': 0.0403, 'grad_norm': 4.27932651497323, 'learning_rate': 9.728395061728396e-06, 'epoch': 0.25}\n",
      "{'loss': 0.0, 'grad_norm': 0.0016430758178673072, 'learning_rate': 9.724867724867725e-06, 'epoch': 0.25}\n",
      "{'loss': 0.0001, 'grad_norm': 0.006668627649691366, 'learning_rate': 9.721340388007056e-06, 'epoch': 0.25}\n",
      "{'loss': 0.0005, 'grad_norm': 0.055018075122055754, 'learning_rate': 9.717813051146385e-06, 'epoch': 0.25}\n",
      "{'loss': 0.0, 'grad_norm': 0.0008825658121589978, 'learning_rate': 9.714285714285715e-06, 'epoch': 0.25}\n",
      "{'loss': 0.0, 'grad_norm': 2.558311643268485e-05, 'learning_rate': 9.710758377425044e-06, 'epoch': 0.25}\n",
      "{'loss': 0.0, 'grad_norm': 1.1245356924660007e-05, 'learning_rate': 9.707231040564375e-06, 'epoch': 0.25}\n",
      "{'loss': 0.0, 'grad_norm': 3.948049230650812e-05, 'learning_rate': 9.703703703703703e-06, 'epoch': 0.25}\n",
      "{'loss': 2.1074, 'grad_norm': 38.409477526610374, 'learning_rate': 9.700176366843034e-06, 'epoch': 0.25}\n",
      "{'loss': 0.0003, 'grad_norm': 0.04511967936978909, 'learning_rate': 9.696649029982365e-06, 'epoch': 0.26}\n",
      "{'loss': 0.0, 'grad_norm': 0.0006174236828837448, 'learning_rate': 9.693121693121693e-06, 'epoch': 0.26}\n",
      "{'loss': 0.0, 'grad_norm': 0.0001031603375190403, 'learning_rate': 9.689594356261024e-06, 'epoch': 0.26}\n",
      "{'loss': 0.0001, 'grad_norm': 0.01170081763335406, 'learning_rate': 9.686067019400353e-06, 'epoch': 0.26}\n",
      "{'loss': 0.0, 'grad_norm': 5.188853674066472e-06, 'learning_rate': 9.682539682539683e-06, 'epoch': 0.26}\n",
      "{'loss': 0.0, 'grad_norm': 0.001013810635372773, 'learning_rate': 9.679012345679012e-06, 'epoch': 0.26}\n",
      "{'loss': 0.0043, 'grad_norm': 0.5165040585782178, 'learning_rate': 9.675485008818343e-06, 'epoch': 0.26}\n",
      "{'loss': 0.0007, 'grad_norm': 0.07455888407813523, 'learning_rate': 9.671957671957672e-06, 'epoch': 0.26}\n",
      "{'loss': 0.0, 'grad_norm': 0.0005316743134375812, 'learning_rate': 9.668430335097002e-06, 'epoch': 0.26}\n",
      "{'loss': 0.0, 'grad_norm': 8.019356688935676e-05, 'learning_rate': 9.664902998236331e-06, 'epoch': 0.26}\n",
      "{'loss': 0.0, 'grad_norm': 0.0007677981148811212, 'learning_rate': 9.661375661375663e-06, 'epoch': 0.26}\n",
      "{'loss': 0.1245, 'grad_norm': 12.541141401847474, 'learning_rate': 9.657848324514992e-06, 'epoch': 0.26}\n",
      "{'loss': 0.0, 'grad_norm': 3.762765089423011e-05, 'learning_rate': 9.654320987654323e-06, 'epoch': 0.26}\n",
      "{'loss': 0.0838, 'grad_norm': 6.604401144042929, 'learning_rate': 9.650793650793652e-06, 'epoch': 0.26}\n",
      "{'loss': 0.0, 'grad_norm': 0.00012301365519960016, 'learning_rate': 9.64726631393298e-06, 'epoch': 0.26}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 13%|█▎        | 417/3150 [01:45<10:51,  4.20it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.0052, 'grad_norm': 0.5387894588544504, 'learning_rate': 9.643738977072311e-06, 'epoch': 0.26}\n",
      "{'loss': 0.002, 'grad_norm': 0.20979235778898053, 'learning_rate': 9.64021164021164e-06, 'epoch': 0.27}\n",
      "{'loss': 0.0002, 'grad_norm': 0.026038436142895877, 'learning_rate': 9.63668430335097e-06, 'epoch': 0.27}\n",
      "{'loss': 0.0, 'grad_norm': 0.00018212249686265307, 'learning_rate': 9.6331569664903e-06, 'epoch': 0.27}\n",
      "{'loss': 0.0083, 'grad_norm': 1.033955002999129, 'learning_rate': 9.62962962962963e-06, 'epoch': 0.27}\n",
      "{'loss': 0.0023, 'grad_norm': 0.343699549093858, 'learning_rate': 9.62610229276896e-06, 'epoch': 0.27}\n",
      "{'loss': 0.0, 'grad_norm': 0.0010749272909065962, 'learning_rate': 9.622574955908291e-06, 'epoch': 0.27}\n",
      "{'loss': 0.0, 'grad_norm': 0.00010975655595019302, 'learning_rate': 9.61904761904762e-06, 'epoch': 0.27}\n",
      "{'loss': 0.0051, 'grad_norm': 0.4788360612721627, 'learning_rate': 9.61552028218695e-06, 'epoch': 0.27}\n",
      "{'loss': 0.0, 'grad_norm': 0.00011672187140924894, 'learning_rate': 9.61199294532628e-06, 'epoch': 0.27}\n",
      "{'loss': 0.0127, 'grad_norm': 1.235076415288614, 'learning_rate': 9.60846560846561e-06, 'epoch': 0.27}\n",
      "{'loss': 0.0631, 'grad_norm': 7.0567203604857225, 'learning_rate': 9.604938271604939e-06, 'epoch': 0.27}\n",
      "{'loss': 0.0765, 'grad_norm': 5.138492594150226, 'learning_rate': 9.60141093474427e-06, 'epoch': 0.27}\n",
      "{'loss': 0.1072, 'grad_norm': 5.9901455380395, 'learning_rate': 9.597883597883598e-06, 'epoch': 0.27}\n",
      "{'loss': 0.0001, 'grad_norm': 0.011286039576731641, 'learning_rate': 9.594356261022927e-06, 'epoch': 0.27}\n",
      "{'loss': 0.0001, 'grad_norm': 0.007324422148026052, 'learning_rate': 9.59082892416226e-06, 'epoch': 0.27}\n",
      "{'loss': 0.0, 'grad_norm': 0.004669892189648679, 'learning_rate': 9.587301587301588e-06, 'epoch': 0.27}\n",
      "{'loss': 0.1426, 'grad_norm': 13.197811199550642, 'learning_rate': 9.583774250440919e-06, 'epoch': 0.28}\n",
      "{'loss': 0.0, 'grad_norm': 0.0046502305784175595, 'learning_rate': 9.580246913580248e-06, 'epoch': 0.28}\n",
      "{'loss': 0.0021, 'grad_norm': 0.223349195650675, 'learning_rate': 9.576719576719578e-06, 'epoch': 0.28}\n",
      "{'loss': 0.0, 'grad_norm': 0.0003301224761112619, 'learning_rate': 9.573192239858907e-06, 'epoch': 0.28}\n",
      "{'loss': 0.0001, 'grad_norm': 0.009009747644794541, 'learning_rate': 9.569664902998238e-06, 'epoch': 0.28}\n",
      "{'loss': 0.0002, 'grad_norm': 0.012643622890240434, 'learning_rate': 9.566137566137567e-06, 'epoch': 0.28}\n",
      "{'loss': 0.0084, 'grad_norm': 0.76595151517225, 'learning_rate': 9.562610229276897e-06, 'epoch': 0.28}\n",
      "{'loss': 0.0051, 'grad_norm': 0.8434381968497825, 'learning_rate': 9.559082892416226e-06, 'epoch': 0.28}\n",
      "{'loss': 0.0, 'grad_norm': 0.0005616020609286258, 'learning_rate': 9.555555555555556e-06, 'epoch': 0.28}\n",
      "{'loss': 0.1694, 'grad_norm': 18.924209412926526, 'learning_rate': 9.552028218694887e-06, 'epoch': 0.28}\n",
      "{'loss': 0.0, 'grad_norm': 0.00010424435317796127, 'learning_rate': 9.548500881834216e-06, 'epoch': 0.28}\n",
      "{'loss': 0.0003, 'grad_norm': 0.043490916339257245, 'learning_rate': 9.544973544973546e-06, 'epoch': 0.28}\n",
      "{'loss': 0.0, 'grad_norm': 6.68762221850048e-05, 'learning_rate': 9.541446208112875e-06, 'epoch': 0.28}\n",
      "{'loss': 0.0, 'grad_norm': 0.0014054202334219392, 'learning_rate': 9.537918871252206e-06, 'epoch': 0.28}\n",
      "{'loss': 0.003, 'grad_norm': 0.35417977009636925, 'learning_rate': 9.534391534391535e-06, 'epoch': 0.28}\n",
      "{'loss': 0.0114, 'grad_norm': 1.6747523708346654, 'learning_rate': 9.530864197530865e-06, 'epoch': 0.29}\n",
      "{'loss': 0.0, 'grad_norm': 2.503336179264217e-06, 'learning_rate': 9.527336860670194e-06, 'epoch': 0.29}\n",
      "{'loss': 0.3484, 'grad_norm': 21.332894956535142, 'learning_rate': 9.523809523809525e-06, 'epoch': 0.29}\n",
      "{'loss': 0.0704, 'grad_norm': 7.407228043115376, 'learning_rate': 9.520282186948855e-06, 'epoch': 0.29}\n",
      "{'loss': 0.2233, 'grad_norm': 10.854382493892984, 'learning_rate': 9.516754850088184e-06, 'epoch': 0.29}\n",
      "{'loss': 0.0, 'grad_norm': 0.0006763692422210834, 'learning_rate': 9.513227513227515e-06, 'epoch': 0.29}\n",
      "{'loss': 0.0009, 'grad_norm': 0.10464323125563814, 'learning_rate': 9.509700176366844e-06, 'epoch': 0.29}\n",
      "{'loss': 0.0112, 'grad_norm': 1.8002737658319494, 'learning_rate': 9.506172839506174e-06, 'epoch': 0.29}\n",
      "{'loss': 0.0003, 'grad_norm': 0.03950459084822429, 'learning_rate': 9.502645502645503e-06, 'epoch': 0.29}\n",
      "{'loss': 0.0081, 'grad_norm': 0.9476139018258405, 'learning_rate': 9.499118165784834e-06, 'epoch': 0.29}\n",
      "{'loss': 0.0, 'grad_norm': 0.0002810139852314601, 'learning_rate': 9.495590828924162e-06, 'epoch': 0.29}\n",
      "{'loss': 0.0, 'grad_norm': 0.003248687324481972, 'learning_rate': 9.492063492063493e-06, 'epoch': 0.29}\n",
      "{'loss': 0.0, 'grad_norm': 1.3370187804656313e-05, 'learning_rate': 9.488536155202822e-06, 'epoch': 0.29}\n",
      "{'loss': 0.2634, 'grad_norm': 22.118658331524514, 'learning_rate': 9.485008818342152e-06, 'epoch': 0.29}\n",
      "{'loss': 0.0, 'grad_norm': 8.84086227984042e-06, 'learning_rate': 9.481481481481483e-06, 'epoch': 0.29}\n",
      "{'loss': 0.0001, 'grad_norm': 0.006092693151434778, 'learning_rate': 9.477954144620812e-06, 'epoch': 0.29}\n",
      "{'loss': 0.0511, 'grad_norm': 4.97184513066285, 'learning_rate': 9.474426807760142e-06, 'epoch': 0.3}\n",
      "{'loss': 0.1105, 'grad_norm': 9.551431470708904, 'learning_rate': 9.470899470899471e-06, 'epoch': 0.3}\n",
      "{'loss': 0.0, 'grad_norm': 0.0010826850979057682, 'learning_rate': 9.467372134038802e-06, 'epoch': 0.3}\n",
      "{'loss': 0.0, 'grad_norm': 0.0016705545965563855, 'learning_rate': 9.46384479717813e-06, 'epoch': 0.3}\n",
      "{'loss': 0.0001, 'grad_norm': 0.005627443940416539, 'learning_rate': 9.460317460317461e-06, 'epoch': 0.3}\n",
      "{'loss': 0.0, 'grad_norm': 0.0017637923522766854, 'learning_rate': 9.45679012345679e-06, 'epoch': 0.3}\n",
      "{'loss': 0.7749, 'grad_norm': 23.789451197563384, 'learning_rate': 9.45326278659612e-06, 'epoch': 0.3}\n",
      "{'loss': 0.0, 'grad_norm': 7.995309096063477e-05, 'learning_rate': 9.449735449735451e-06, 'epoch': 0.3}\n",
      "{'loss': 0.0, 'grad_norm': 0.00038951766244453887, 'learning_rate': 9.44620811287478e-06, 'epoch': 0.3}\n",
      "{'loss': 0.0007, 'grad_norm': 0.06236068464811764, 'learning_rate': 9.44268077601411e-06, 'epoch': 0.3}\n",
      "{'loss': 0.0, 'grad_norm': 0.001694052708670376, 'learning_rate': 9.43915343915344e-06, 'epoch': 0.3}\n",
      "{'loss': 0.0, 'grad_norm': 0.000439431782319481, 'learning_rate': 9.43562610229277e-06, 'epoch': 0.3}\n",
      "{'loss': 2.0938, 'grad_norm': 32.86156859356667, 'learning_rate': 9.432098765432099e-06, 'epoch': 0.3}\n",
      "{'loss': 0.0, 'grad_norm': 2.6667594096080437e-05, 'learning_rate': 9.42857142857143e-06, 'epoch': 0.3}\n",
      "{'loss': 0.0023, 'grad_norm': 0.3417198966399091, 'learning_rate': 9.425044091710758e-06, 'epoch': 0.3}\n",
      "{'loss': 0.0, 'grad_norm': 0.00011078894598663748, 'learning_rate': 9.421516754850089e-06, 'epoch': 0.3}\n",
      "{'loss': 0.0623, 'grad_norm': 7.977362552464131, 'learning_rate': 9.417989417989418e-06, 'epoch': 0.31}\n",
      "{'loss': 0.0001, 'grad_norm': 0.02046773391692334, 'learning_rate': 9.414462081128748e-06, 'epoch': 0.31}\n",
      "{'loss': 0.0, 'grad_norm': 0.00022039384629892231, 'learning_rate': 9.410934744268079e-06, 'epoch': 0.31}\n",
      "{'loss': 0.0001, 'grad_norm': 0.00639420501258587, 'learning_rate': 9.407407407407408e-06, 'epoch': 0.31}\n",
      "{'loss': 0.0002, 'grad_norm': 0.01871886291200188, 'learning_rate': 9.403880070546738e-06, 'epoch': 0.31}\n",
      "{'loss': 0.0, 'grad_norm': 0.0010791559716901737, 'learning_rate': 9.400352733686067e-06, 'epoch': 0.31}\n",
      "{'loss': 0.0025, 'grad_norm': 0.31608098583689, 'learning_rate': 9.396825396825398e-06, 'epoch': 0.31}\n",
      "{'loss': 0.0005, 'grad_norm': 0.05519794972444185, 'learning_rate': 9.393298059964727e-06, 'epoch': 0.31}\n",
      "{'loss': 0.001, 'grad_norm': 0.08670803091884981, 'learning_rate': 9.389770723104057e-06, 'epoch': 0.31}\n",
      "{'loss': 0.0052, 'grad_norm': 0.5841564065419473, 'learning_rate': 9.386243386243386e-06, 'epoch': 0.31}\n",
      "{'loss': 0.0846, 'grad_norm': 7.669195200488338, 'learning_rate': 9.382716049382717e-06, 'epoch': 0.31}\n",
      "{'loss': 0.0, 'grad_norm': 0.0001556751232194212, 'learning_rate': 9.379188712522047e-06, 'epoch': 0.31}\n",
      "{'loss': 0.0008, 'grad_norm': 0.12388771620843084, 'learning_rate': 9.375661375661376e-06, 'epoch': 0.31}\n",
      "{'loss': 0.0003, 'grad_norm': 0.034952368898296775, 'learning_rate': 9.372134038800707e-06, 'epoch': 0.31}\n",
      "{'loss': 0.0, 'grad_norm': 1.3114840978388855e-05, 'learning_rate': 9.368606701940036e-06, 'epoch': 0.31}\n",
      "{'loss': 0.0, 'grad_norm': 0.0010697442324237544, 'learning_rate': 9.365079365079366e-06, 'epoch': 0.31}\n",
      "{'loss': 0.0001, 'grad_norm': 0.0082480794657335, 'learning_rate': 9.361552028218695e-06, 'epoch': 0.32}\n",
      "{'loss': 0.0008, 'grad_norm': 0.10762173094392417, 'learning_rate': 9.358024691358025e-06, 'epoch': 0.32}\n",
      "{'loss': 0.0, 'grad_norm': 0.00028685552566825256, 'learning_rate': 9.354497354497354e-06, 'epoch': 0.32}\n",
      "{'loss': 0.0002, 'grad_norm': 0.02336619378084213, 'learning_rate': 9.350970017636685e-06, 'epoch': 0.32}\n",
      "{'loss': 0.5278, 'grad_norm': 25.117641908797765, 'learning_rate': 9.347442680776014e-06, 'epoch': 0.32}\n",
      "{'loss': 0.0, 'grad_norm': 0.0005722173301947363, 'learning_rate': 9.343915343915344e-06, 'epoch': 0.32}\n",
      "{'loss': 0.4949, 'grad_norm': 17.54948621905309, 'learning_rate': 9.340388007054675e-06, 'epoch': 0.32}\n",
      "{'loss': 0.0013, 'grad_norm': 0.14646107170394695, 'learning_rate': 9.336860670194004e-06, 'epoch': 0.32}\n",
      "{'loss': 0.0, 'grad_norm': 0.0013985811057465423, 'learning_rate': 9.333333333333334e-06, 'epoch': 0.32}\n",
      "{'loss': 0.5957, 'grad_norm': 31.228124179112868, 'learning_rate': 9.329805996472663e-06, 'epoch': 0.32}\n",
      "{'loss': 0.0001, 'grad_norm': 0.012013351442862003, 'learning_rate': 9.326278659611994e-06, 'epoch': 0.32}\n",
      "{'loss': 0.0009, 'grad_norm': 0.07082839295862332, 'learning_rate': 9.322751322751323e-06, 'epoch': 0.32}\n",
      "{'loss': 0.0005, 'grad_norm': 0.05599493478920188, 'learning_rate': 9.319223985890653e-06, 'epoch': 0.32}\n",
      "{'loss': 0.0003, 'grad_norm': 0.03774758424385116, 'learning_rate': 9.315696649029982e-06, 'epoch': 0.32}\n",
      "{'loss': 0.1407, 'grad_norm': 9.952884689136809, 'learning_rate': 9.312169312169313e-06, 'epoch': 0.32}\n",
      "{'loss': 0.0, 'grad_norm': 2.9845431002689647e-05, 'learning_rate': 9.308641975308643e-06, 'epoch': 0.33}\n",
      "{'loss': 0.0006, 'grad_norm': 0.0637212548652235, 'learning_rate': 9.305114638447974e-06, 'epoch': 0.33}\n",
      "{'loss': 0.0, 'grad_norm': 0.0002740915166792806, 'learning_rate': 9.301587301587303e-06, 'epoch': 0.33}\n",
      "{'loss': 0.0, 'grad_norm': 0.0032825947912661162, 'learning_rate': 9.298059964726633e-06, 'epoch': 0.33}\n",
      "{'loss': 0.0107, 'grad_norm': 1.239868061906578, 'learning_rate': 9.294532627865962e-06, 'epoch': 0.33}\n",
      "{'loss': 1.4463, 'grad_norm': 39.02784400171082, 'learning_rate': 9.291005291005291e-06, 'epoch': 0.33}\n",
      "{'loss': 0.2534, 'grad_norm': 13.572009150197582, 'learning_rate': 9.287477954144621e-06, 'epoch': 0.33}\n",
      "{'loss': 0.0, 'grad_norm': 0.0008117765047677844, 'learning_rate': 9.28395061728395e-06, 'epoch': 0.33}\n",
      "{'loss': 0.1361, 'grad_norm': 10.907640603501576, 'learning_rate': 9.280423280423281e-06, 'epoch': 0.33}\n",
      "{'loss': 0.1516, 'grad_norm': 16.006840197411503, 'learning_rate': 9.27689594356261e-06, 'epoch': 0.33}\n",
      "{'loss': 0.0003, 'grad_norm': 0.0326651847488497, 'learning_rate': 9.273368606701942e-06, 'epoch': 0.33}\n",
      "{'loss': 0.0, 'grad_norm': 0.00041121986998010195, 'learning_rate': 9.26984126984127e-06, 'epoch': 0.33}\n",
      "{'loss': 0.0004, 'grad_norm': 0.04407502343255406, 'learning_rate': 9.266313932980601e-06, 'epoch': 0.33}\n",
      "{'loss': 0.0003, 'grad_norm': 0.022330090878106876, 'learning_rate': 9.26278659611993e-06, 'epoch': 0.33}\n",
      "{'loss': 0.0, 'grad_norm': 0.0018842050610009468, 'learning_rate': 9.25925925925926e-06, 'epoch': 0.33}\n",
      "{'loss': 0.0001, 'grad_norm': 0.009732819007203106, 'learning_rate': 9.25573192239859e-06, 'epoch': 0.33}\n",
      "{'loss': 0.0, 'grad_norm': 0.0008740279104904014, 'learning_rate': 9.25220458553792e-06, 'epoch': 0.34}\n",
      "{'loss': 0.0005, 'grad_norm': 0.04903269116199547, 'learning_rate': 9.248677248677249e-06, 'epoch': 0.34}\n",
      "{'loss': 0.0002, 'grad_norm': 0.02098511530822922, 'learning_rate': 9.24514991181658e-06, 'epoch': 0.34}\n",
      "{'loss': 0.0, 'grad_norm': 0.003808187533189958, 'learning_rate': 9.241622574955909e-06, 'epoch': 0.34}\n",
      "{'loss': 0.004, 'grad_norm': 0.3289230722941331, 'learning_rate': 9.238095238095239e-06, 'epoch': 0.34}\n",
      "{'loss': 0.0003, 'grad_norm': 0.03945174740048061, 'learning_rate': 9.23456790123457e-06, 'epoch': 0.34}\n",
      "{'loss': 0.0, 'grad_norm': 0.001456448343672392, 'learning_rate': 9.231040564373899e-06, 'epoch': 0.34}\n",
      "{'loss': 0.0008, 'grad_norm': 0.07375150973989174, 'learning_rate': 9.227513227513229e-06, 'epoch': 0.34}\n",
      "{'loss': 0.0, 'grad_norm': 0.00012353433984781679, 'learning_rate': 9.223985890652558e-06, 'epoch': 0.34}\n",
      "{'loss': 0.0003, 'grad_norm': 0.03674043284052274, 'learning_rate': 9.220458553791889e-06, 'epoch': 0.34}\n",
      "{'loss': 0.0001, 'grad_norm': 0.012973167922937038, 'learning_rate': 9.216931216931217e-06, 'epoch': 0.34}\n",
      "{'loss': 0.0993, 'grad_norm': 7.095295338629874, 'learning_rate': 9.213403880070548e-06, 'epoch': 0.34}\n",
      "{'loss': 0.3013, 'grad_norm': 7.94463958441292, 'learning_rate': 9.209876543209877e-06, 'epoch': 0.34}\n",
      "{'loss': 0.0001, 'grad_norm': 0.010281723047042193, 'learning_rate': 9.206349206349207e-06, 'epoch': 0.34}\n",
      "{'loss': 0.0033, 'grad_norm': 0.29479833882917134, 'learning_rate': 9.202821869488538e-06, 'epoch': 0.34}\n",
      "{'loss': 0.0015, 'grad_norm': 0.11959528158541345, 'learning_rate': 9.199294532627867e-06, 'epoch': 0.34}\n",
      "{'loss': 0.0121, 'grad_norm': 1.458181709400027, 'learning_rate': 9.195767195767197e-06, 'epoch': 0.35}\n",
      "{'loss': 0.0, 'grad_norm': 0.002892840842918088, 'learning_rate': 9.192239858906526e-06, 'epoch': 0.35}\n",
      "{'loss': 0.0, 'grad_norm': 0.003803119733606415, 'learning_rate': 9.188712522045857e-06, 'epoch': 0.35}\n",
      "{'loss': 0.0005, 'grad_norm': 0.06089786155812499, 'learning_rate': 9.185185185185186e-06, 'epoch': 0.35}\n",
      "{'loss': 0.0001, 'grad_norm': 0.007054187290533145, 'learning_rate': 9.181657848324516e-06, 'epoch': 0.35}\n",
      "{'loss': 0.0004, 'grad_norm': 0.03912794620921914, 'learning_rate': 9.178130511463845e-06, 'epoch': 0.35}\n",
      "{'loss': 0.0, 'grad_norm': 0.0008316480568604834, 'learning_rate': 9.174603174603176e-06, 'epoch': 0.35}\n",
      "{'loss': 0.0, 'grad_norm': 2.8363559877941435e-05, 'learning_rate': 9.171075837742504e-06, 'epoch': 0.35}\n",
      "{'loss': 0.0, 'grad_norm': 0.0005230903625488281, 'learning_rate': 9.167548500881835e-06, 'epoch': 0.35}\n",
      "{'loss': 0.0023, 'grad_norm': 0.23891687040255982, 'learning_rate': 9.164021164021166e-06, 'epoch': 0.35}\n",
      "{'loss': 0.0, 'grad_norm': 0.0003531969074626214, 'learning_rate': 9.160493827160494e-06, 'epoch': 0.35}\n",
      "{'loss': 0.2019, 'grad_norm': 22.963392232257412, 'learning_rate': 9.156966490299825e-06, 'epoch': 0.35}\n",
      "{'loss': 0.0003, 'grad_norm': 0.039030843066422266, 'learning_rate': 9.153439153439154e-06, 'epoch': 0.35}\n",
      "{'loss': 0.0073, 'grad_norm': 0.5546233180349216, 'learning_rate': 9.149911816578484e-06, 'epoch': 0.35}\n",
      "{'loss': 0.0, 'grad_norm': 0.0003821190694214784, 'learning_rate': 9.146384479717813e-06, 'epoch': 0.35}\n",
      "{'loss': 0.0015, 'grad_norm': 0.1884442138582916, 'learning_rate': 9.142857142857144e-06, 'epoch': 0.35}\n",
      "{'loss': 0.0287, 'grad_norm': 3.251491791131857, 'learning_rate': 9.139329805996473e-06, 'epoch': 0.36}\n",
      "{'loss': 0.0001, 'grad_norm': 0.016972435681201558, 'learning_rate': 9.135802469135803e-06, 'epoch': 0.36}\n",
      "{'loss': 0.004, 'grad_norm': 0.43338665871002285, 'learning_rate': 9.132275132275134e-06, 'epoch': 0.36}\n",
      "{'loss': 0.0009, 'grad_norm': 0.1150372241121039, 'learning_rate': 9.128747795414463e-06, 'epoch': 0.36}\n",
      "{'loss': 0.0029, 'grad_norm': 0.2843923002786516, 'learning_rate': 9.125220458553793e-06, 'epoch': 0.36}\n",
      "{'loss': 0.0003, 'grad_norm': 0.02143423436497846, 'learning_rate': 9.121693121693122e-06, 'epoch': 0.36}\n",
      "{'loss': 1.8467, 'grad_norm': 36.21284127277056, 'learning_rate': 9.118165784832453e-06, 'epoch': 0.36}\n",
      "{'loss': 0.0095, 'grad_norm': 1.1975148674314742, 'learning_rate': 9.114638447971782e-06, 'epoch': 0.36}\n",
      "{'loss': 0.0, 'grad_norm': 0.00026924198555036287, 'learning_rate': 9.111111111111112e-06, 'epoch': 0.36}\n",
      "{'loss': 0.0, 'grad_norm': 0.005284171953739357, 'learning_rate': 9.107583774250441e-06, 'epoch': 0.36}\n",
      "{'loss': 0.0, 'grad_norm': 0.0003040957127578065, 'learning_rate': 9.104056437389772e-06, 'epoch': 0.36}\n",
      "{'loss': 0.0, 'grad_norm': 2.0002015086789646e-05, 'learning_rate': 9.1005291005291e-06, 'epoch': 0.36}\n",
      "{'loss': 0.4585, 'grad_norm': 12.468772027706107, 'learning_rate': 9.097001763668431e-06, 'epoch': 0.36}\n",
      "{'loss': 0.1539, 'grad_norm': 11.185247269047142, 'learning_rate': 9.093474426807762e-06, 'epoch': 0.36}\n",
      "{'loss': 0.0, 'grad_norm': 0.00021001007869739246, 'learning_rate': 9.08994708994709e-06, 'epoch': 0.36}\n",
      "{'loss': 0.0, 'grad_norm': 1.8440867300681122e-05, 'learning_rate': 9.086419753086421e-06, 'epoch': 0.37}\n",
      "{'loss': 0.0, 'grad_norm': 7.081352580937269e-05, 'learning_rate': 9.08289241622575e-06, 'epoch': 0.37}\n",
      "{'loss': 0.0, 'grad_norm': 0.0012116788264405357, 'learning_rate': 9.07936507936508e-06, 'epoch': 0.37}\n",
      "{'loss': 0.0031, 'grad_norm': 0.3480657887055981, 'learning_rate': 9.07583774250441e-06, 'epoch': 0.37}\n",
      "{'loss': 0.0, 'grad_norm': 0.0025071252552526205, 'learning_rate': 9.07231040564374e-06, 'epoch': 0.37}\n",
      "{'loss': 0.0161, 'grad_norm': 1.5625627123645378, 'learning_rate': 9.068783068783069e-06, 'epoch': 0.37}\n",
      "{'loss': 0.004, 'grad_norm': 0.4820900830142812, 'learning_rate': 9.0652557319224e-06, 'epoch': 0.37}\n",
      "{'loss': 0.0, 'grad_norm': 0.001218707438895755, 'learning_rate': 9.06172839506173e-06, 'epoch': 0.37}\n",
      "{'loss': 0.8765, 'grad_norm': 23.033980767976935, 'learning_rate': 9.058201058201059e-06, 'epoch': 0.37}\n",
      "{'loss': 0.0001, 'grad_norm': 0.008424474984744178, 'learning_rate': 9.05467372134039e-06, 'epoch': 0.37}\n",
      "{'loss': 0.0012, 'grad_norm': 0.13757991689958848, 'learning_rate': 9.051146384479718e-06, 'epoch': 0.37}\n",
      "{'loss': 0.0, 'grad_norm': 0.0003590236882311191, 'learning_rate': 9.047619047619049e-06, 'epoch': 0.37}\n",
      "{'loss': 0.6885, 'grad_norm': 26.055480450827183, 'learning_rate': 9.044091710758378e-06, 'epoch': 0.37}\n",
      "{'loss': 0.003, 'grad_norm': 0.4125008554160756, 'learning_rate': 9.040564373897708e-06, 'epoch': 0.37}\n",
      "{'loss': 0.2177, 'grad_norm': 21.233152644867282, 'learning_rate': 9.037037037037037e-06, 'epoch': 0.37}\n",
      "{'loss': 0.0, 'grad_norm': 0.004493951791160916, 'learning_rate': 9.033509700176368e-06, 'epoch': 0.37}\n",
      "{'loss': 0.0, 'grad_norm': 0.0013479146978041194, 'learning_rate': 9.029982363315696e-06, 'epoch': 0.38}\n",
      "{'loss': 0.0, 'grad_norm': 0.001465321440592504, 'learning_rate': 9.026455026455027e-06, 'epoch': 0.38}\n",
      "{'loss': 0.0, 'grad_norm': 0.00027815552126746456, 'learning_rate': 9.022927689594358e-06, 'epoch': 0.38}\n",
      "{'loss': 0.5859, 'grad_norm': 38.853767312514236, 'learning_rate': 9.019400352733686e-06, 'epoch': 0.38}\n",
      "{'loss': 0.0015, 'grad_norm': 0.19520796839565605, 'learning_rate': 9.015873015873017e-06, 'epoch': 0.38}\n",
      "{'loss': 0.0, 'grad_norm': 0.0008940756161810027, 'learning_rate': 9.012345679012346e-06, 'epoch': 0.38}\n",
      "{'loss': 1.1172, 'grad_norm': 25.76404136847942, 'learning_rate': 9.008818342151676e-06, 'epoch': 0.38}\n",
      "{'loss': 0.0, 'grad_norm': 0.004197920302910475, 'learning_rate': 9.005291005291005e-06, 'epoch': 0.38}\n",
      "{'loss': 0.0, 'grad_norm': 0.0007062849307986124, 'learning_rate': 9.001763668430336e-06, 'epoch': 0.38}\n",
      "{'loss': 0.0, 'grad_norm': 0.0020492712508102585, 'learning_rate': 8.998236331569665e-06, 'epoch': 0.38}\n",
      "{'loss': 0.0, 'grad_norm': 2.6128567919991973e-05, 'learning_rate': 8.994708994708995e-06, 'epoch': 0.38}\n",
      "{'loss': 0.0002, 'grad_norm': 0.021187182331807084, 'learning_rate': 8.991181657848326e-06, 'epoch': 0.38}\n",
      "{'loss': 0.0164, 'grad_norm': 1.5493341800164313, 'learning_rate': 8.987654320987655e-06, 'epoch': 0.38}\n",
      "{'loss': 0.0006, 'grad_norm': 0.06926588764437759, 'learning_rate': 8.984126984126985e-06, 'epoch': 0.38}\n",
      "{'loss': 0.0, 'grad_norm': 0.0003429978997751221, 'learning_rate': 8.980599647266314e-06, 'epoch': 0.38}\n",
      "{'loss': 0.0169, 'grad_norm': 2.3904497668300912, 'learning_rate': 8.977072310405645e-06, 'epoch': 0.38}\n",
      "{'loss': 0.0, 'grad_norm': 2.2668978872834254e-05, 'learning_rate': 8.973544973544973e-06, 'epoch': 0.39}\n",
      "{'loss': 0.0468, 'grad_norm': 5.606426175856776, 'learning_rate': 8.970017636684304e-06, 'epoch': 0.39}\n",
      "{'loss': 0.0064, 'grad_norm': 0.7068214315932386, 'learning_rate': 8.966490299823633e-06, 'epoch': 0.39}\n",
      "{'loss': 0.0, 'grad_norm': 0.0001739955958565241, 'learning_rate': 8.962962962962963e-06, 'epoch': 0.39}\n",
      "{'loss': 0.0, 'grad_norm': 8.368342211374271e-05, 'learning_rate': 8.959435626102292e-06, 'epoch': 0.39}\n",
      "{'loss': 0.0001, 'grad_norm': 0.006923836582717384, 'learning_rate': 8.955908289241625e-06, 'epoch': 0.39}\n",
      "{'loss': 0.0, 'grad_norm': 0.0007167858605312458, 'learning_rate': 8.952380952380953e-06, 'epoch': 0.39}\n",
      "{'loss': 0.0002, 'grad_norm': 0.017944864667511756, 'learning_rate': 8.948853615520284e-06, 'epoch': 0.39}\n",
      "{'loss': 0.1708, 'grad_norm': 10.272358768817071, 'learning_rate': 8.945326278659613e-06, 'epoch': 0.39}\n",
      "{'loss': 0.0008, 'grad_norm': 0.10433918455520556, 'learning_rate': 8.941798941798942e-06, 'epoch': 0.39}\n",
      "{'loss': 0.0013, 'grad_norm': 0.24473960640769105, 'learning_rate': 8.938271604938272e-06, 'epoch': 0.39}\n",
      "{'loss': 0.77, 'grad_norm': 29.296085406026307, 'learning_rate': 8.934744268077601e-06, 'epoch': 0.39}\n",
      "{'loss': 0.0474, 'grad_norm': 5.879123295931101, 'learning_rate': 8.931216931216932e-06, 'epoch': 0.39}\n",
      "{'loss': 0.0041, 'grad_norm': 0.34788167267354597, 'learning_rate': 8.92768959435626e-06, 'epoch': 0.39}\n",
      "{'loss': 0.0001, 'grad_norm': 0.005503977622482464, 'learning_rate': 8.924162257495591e-06, 'epoch': 0.39}\n",
      "{'loss': 0.0005, 'grad_norm': 0.05812897927816892, 'learning_rate': 8.920634920634922e-06, 'epoch': 0.39}\n",
      "{'loss': 0.0002, 'grad_norm': 0.017057393123176952, 'learning_rate': 8.917107583774252e-06, 'epoch': 0.4}\n",
      "{'loss': 0.0001, 'grad_norm': 0.011785971297019419, 'learning_rate': 8.913580246913581e-06, 'epoch': 0.4}\n",
      "{'loss': 0.3032, 'grad_norm': 18.870777188273227, 'learning_rate': 8.910052910052912e-06, 'epoch': 0.4}\n",
      "{'loss': 0.0018, 'grad_norm': 0.2118400761001139, 'learning_rate': 8.90652557319224e-06, 'epoch': 0.4}\n",
      "{'loss': 0.0013, 'grad_norm': 0.14888039194291128, 'learning_rate': 8.902998236331571e-06, 'epoch': 0.4}\n",
      "{'loss': 0.0029, 'grad_norm': 0.3198094954268452, 'learning_rate': 8.8994708994709e-06, 'epoch': 0.4}\n",
      "{'loss': 0.0002, 'grad_norm': 0.02115185756424001, 'learning_rate': 8.89594356261023e-06, 'epoch': 0.4}\n",
      "{'loss': 0.0, 'grad_norm': 0.00016344557090485282, 'learning_rate': 8.89241622574956e-06, 'epoch': 0.4}\n",
      "{'loss': 0.0, 'grad_norm': 0.003896253897637417, 'learning_rate': 8.888888888888888e-06, 'epoch': 0.4}\n",
      "{'loss': 0.001, 'grad_norm': 0.13779857497026907, 'learning_rate': 8.88536155202822e-06, 'epoch': 0.4}\n",
      "{'loss': 0.0002, 'grad_norm': 0.021783838004051904, 'learning_rate': 8.88183421516755e-06, 'epoch': 0.4}\n",
      "{'loss': 0.0053, 'grad_norm': 0.6245147490260117, 'learning_rate': 8.87830687830688e-06, 'epoch': 0.4}\n",
      "{'loss': 0.0002, 'grad_norm': 0.0154813706686109, 'learning_rate': 8.874779541446209e-06, 'epoch': 0.4}\n",
      "{'loss': 0.0, 'grad_norm': 4.120536711663732e-05, 'learning_rate': 8.87125220458554e-06, 'epoch': 0.4}\n",
      "{'loss': 0.0, 'grad_norm': 7.248387693988586e-05, 'learning_rate': 8.867724867724868e-06, 'epoch': 0.4}\n",
      "{'loss': 0.0, 'grad_norm': 0.00016204137956638305, 'learning_rate': 8.864197530864199e-06, 'epoch': 0.41}\n",
      "{'loss': 0.0014, 'grad_norm': 0.15313644682663977, 'learning_rate': 8.860670194003528e-06, 'epoch': 0.41}\n",
      "{'loss': 0.0284, 'grad_norm': 1.8067811653528705, 'learning_rate': 8.857142857142858e-06, 'epoch': 0.41}\n",
      "{'loss': 0.1283, 'grad_norm': 12.721806489314421, 'learning_rate': 8.853615520282187e-06, 'epoch': 0.41}\n",
      "{'loss': 0.0, 'grad_norm': 0.0011624943236735643, 'learning_rate': 8.850088183421518e-06, 'epoch': 0.41}\n",
      "{'loss': 0.0008, 'grad_norm': 0.09102052237469278, 'learning_rate': 8.846560846560848e-06, 'epoch': 0.41}\n",
      "{'loss': 0.0016, 'grad_norm': 0.1617842193896177, 'learning_rate': 8.843033509700177e-06, 'epoch': 0.41}\n",
      "{'loss': 1.0879, 'grad_norm': 33.26716618083249, 'learning_rate': 8.839506172839508e-06, 'epoch': 0.41}\n",
      "{'loss': 0.0011, 'grad_norm': 0.14618742407992097, 'learning_rate': 8.835978835978837e-06, 'epoch': 0.41}\n",
      "{'loss': 0.0, 'grad_norm': 6.098596732647754e-05, 'learning_rate': 8.832451499118167e-06, 'epoch': 0.41}\n",
      "{'loss': 0.0, 'grad_norm': 0.0008412504205761954, 'learning_rate': 8.828924162257496e-06, 'epoch': 0.41}\n",
      "{'loss': 0.0017, 'grad_norm': 0.12867445045075537, 'learning_rate': 8.825396825396827e-06, 'epoch': 0.41}\n",
      "{'loss': 0.0, 'grad_norm': 0.0018560069126813386, 'learning_rate': 8.821869488536155e-06, 'epoch': 0.41}\n",
      "{'loss': 0.0356, 'grad_norm': 4.21275095206285, 'learning_rate': 8.818342151675486e-06, 'epoch': 0.41}\n",
      "{'loss': 0.0, 'grad_norm': 0.004305533147917828, 'learning_rate': 8.814814814814817e-06, 'epoch': 0.41}\n",
      "{'loss': 0.0167, 'grad_norm': 1.7210581453288152, 'learning_rate': 8.811287477954145e-06, 'epoch': 0.41}\n",
      "{'loss': 0.0, 'grad_norm': 0.0005098653363148452, 'learning_rate': 8.807760141093476e-06, 'epoch': 0.42}\n",
      "{'loss': 0.0, 'grad_norm': 0.0013374830517134556, 'learning_rate': 8.804232804232805e-06, 'epoch': 0.42}\n",
      "{'loss': 0.3474, 'grad_norm': 18.552911921622556, 'learning_rate': 8.800705467372135e-06, 'epoch': 0.42}\n",
      "{'loss': 0.0, 'grad_norm': 0.0007688675363508784, 'learning_rate': 8.797178130511464e-06, 'epoch': 0.42}\n",
      "{'loss': 0.0003, 'grad_norm': 0.03783461441049915, 'learning_rate': 8.793650793650795e-06, 'epoch': 0.42}\n",
      "{'loss': 0.0, 'grad_norm': 7.273069786736758e-05, 'learning_rate': 8.790123456790124e-06, 'epoch': 0.42}\n",
      "{'loss': 0.161, 'grad_norm': 18.870431915660905, 'learning_rate': 8.786596119929454e-06, 'epoch': 0.42}\n",
      "{'loss': 0.0001, 'grad_norm': 0.010706521628314147, 'learning_rate': 8.783068783068783e-06, 'epoch': 0.42}\n",
      "{'loss': 0.0348, 'grad_norm': 3.132434089618753, 'learning_rate': 8.779541446208114e-06, 'epoch': 0.42}\n",
      "{'loss': 0.0, 'grad_norm': 0.00042336916949563535, 'learning_rate': 8.776014109347444e-06, 'epoch': 0.42}\n",
      "{'loss': 0.0006, 'grad_norm': 0.072778553596028, 'learning_rate': 8.772486772486773e-06, 'epoch': 0.42}\n",
      "{'loss': 0.4312, 'grad_norm': 30.225831520626215, 'learning_rate': 8.768959435626104e-06, 'epoch': 0.42}\n",
      "{'loss': 0.0, 'grad_norm': 0.0015560784143454917, 'learning_rate': 8.765432098765432e-06, 'epoch': 0.42}\n",
      "{'loss': 0.0004, 'grad_norm': 0.03230311772489046, 'learning_rate': 8.761904761904763e-06, 'epoch': 0.42}\n",
      "{'loss': 0.0008, 'grad_norm': 0.08954097169918082, 'learning_rate': 8.758377425044092e-06, 'epoch': 0.42}\n",
      "{'loss': 0.0105, 'grad_norm': 1.017326338473563, 'learning_rate': 8.754850088183422e-06, 'epoch': 0.42}\n",
      "{'loss': 0.0106, 'grad_norm': 0.9836870544634239, 'learning_rate': 8.751322751322751e-06, 'epoch': 0.43}\n",
      "{'loss': 0.0, 'grad_norm': 0.003229351200233285, 'learning_rate': 8.747795414462082e-06, 'epoch': 0.43}\n",
      "{'loss': 0.0003, 'grad_norm': 0.04249211403843371, 'learning_rate': 8.744268077601412e-06, 'epoch': 0.43}\n",
      "{'loss': 0.0015, 'grad_norm': 0.263620795494509, 'learning_rate': 8.740740740740741e-06, 'epoch': 0.43}\n",
      "{'loss': 0.0, 'grad_norm': 6.9247402669845175e-06, 'learning_rate': 8.737213403880072e-06, 'epoch': 0.43}\n",
      "{'loss': 0.0227, 'grad_norm': 2.3725958250288, 'learning_rate': 8.7336860670194e-06, 'epoch': 0.43}\n",
      "{'loss': 0.0037, 'grad_norm': 0.36718031693595365, 'learning_rate': 8.730158730158731e-06, 'epoch': 0.43}\n",
      "{'loss': 0.8545, 'grad_norm': 23.055614074060816, 'learning_rate': 8.72663139329806e-06, 'epoch': 0.43}\n",
      "{'loss': 0.259, 'grad_norm': 13.383620291769256, 'learning_rate': 8.72310405643739e-06, 'epoch': 0.43}\n",
      "{'loss': 0.0068, 'grad_norm': 0.9910191242305099, 'learning_rate': 8.71957671957672e-06, 'epoch': 0.43}\n",
      "{'loss': 0.0072, 'grad_norm': 1.1333621566043073, 'learning_rate': 8.71604938271605e-06, 'epoch': 0.43}\n",
      "{'loss': 0.0, 'grad_norm': 0.000716941040113439, 'learning_rate': 8.712522045855379e-06, 'epoch': 0.43}\n",
      "{'loss': 0.0, 'grad_norm': 0.0013733766946291276, 'learning_rate': 8.70899470899471e-06, 'epoch': 0.43}\n",
      "{'loss': 0.001, 'grad_norm': 0.14866676945221274, 'learning_rate': 8.70546737213404e-06, 'epoch': 0.43}\n",
      "{'loss': 0.0001, 'grad_norm': 0.007398283022816927, 'learning_rate': 8.701940035273369e-06, 'epoch': 0.43}\n",
      "{'loss': 0.0003, 'grad_norm': 0.02963145850133267, 'learning_rate': 8.6984126984127e-06, 'epoch': 0.43}\n",
      "{'loss': 0.0001, 'grad_norm': 0.004225600573562544, 'learning_rate': 8.694885361552028e-06, 'epoch': 0.44}\n",
      "{'loss': 0.0, 'grad_norm': 0.0002106226924693437, 'learning_rate': 8.691358024691359e-06, 'epoch': 0.44}\n",
      "{'loss': 0.0, 'grad_norm': 0.0002608405767895165, 'learning_rate': 8.687830687830688e-06, 'epoch': 0.44}\n",
      "{'loss': 0.0, 'grad_norm': 0.00016234621383929936, 'learning_rate': 8.684303350970018e-06, 'epoch': 0.44}\n",
      "{'loss': 0.0, 'grad_norm': 0.00016341901233497624, 'learning_rate': 8.680776014109347e-06, 'epoch': 0.44}\n",
      "{'loss': 0.0, 'grad_norm': 2.3207797941936213e-05, 'learning_rate': 8.677248677248678e-06, 'epoch': 0.44}\n",
      "{'loss': 0.0006, 'grad_norm': 0.06888668834165373, 'learning_rate': 8.673721340388008e-06, 'epoch': 0.44}\n",
      "{'loss': 0.0004, 'grad_norm': 0.0377074996488796, 'learning_rate': 8.670194003527337e-06, 'epoch': 0.44}\n",
      "{'loss': 0.0022, 'grad_norm': 0.27235415974439997, 'learning_rate': 8.666666666666668e-06, 'epoch': 0.44}\n",
      "{'loss': 0.2048, 'grad_norm': 19.55352557849423, 'learning_rate': 8.663139329805997e-06, 'epoch': 0.44}\n",
      "{'loss': 0.0, 'grad_norm': 7.105547997844708e-07, 'learning_rate': 8.659611992945327e-06, 'epoch': 0.44}\n",
      "{'loss': 0.0708, 'grad_norm': 6.6357007247830815, 'learning_rate': 8.656084656084656e-06, 'epoch': 0.44}\n",
      "{'loss': 0.0012, 'grad_norm': 0.10043246866661085, 'learning_rate': 8.652557319223987e-06, 'epoch': 0.44}\n",
      "{'loss': 0.562, 'grad_norm': 20.21049507874869, 'learning_rate': 8.649029982363316e-06, 'epoch': 0.44}\n",
      "{'loss': 2.0391, 'grad_norm': 37.45004648121698, 'learning_rate': 8.645502645502646e-06, 'epoch': 0.44}\n",
      "{'loss': 0.0003, 'grad_norm': 0.036824255708578994, 'learning_rate': 8.641975308641975e-06, 'epoch': 0.45}\n",
      "{'loss': 0.0001, 'grad_norm': 0.014301841361321775, 'learning_rate': 8.638447971781306e-06, 'epoch': 0.45}\n",
      "{'loss': 0.0001, 'grad_norm': 0.004445269645297784, 'learning_rate': 8.634920634920636e-06, 'epoch': 0.45}\n",
      "{'loss': 0.0, 'grad_norm': 0.0004544745623746442, 'learning_rate': 8.631393298059965e-06, 'epoch': 0.45}\n",
      "{'loss': 0.0222, 'grad_norm': 2.2433696116234323, 'learning_rate': 8.627865961199296e-06, 'epoch': 0.45}\n",
      "{'loss': 0.0005, 'grad_norm': 0.0760574122706384, 'learning_rate': 8.624338624338624e-06, 'epoch': 0.45}\n",
      "{'loss': 0.0002, 'grad_norm': 0.016113381493863484, 'learning_rate': 8.620811287477955e-06, 'epoch': 0.45}\n",
      "{'loss': 0.0152, 'grad_norm': 1.80880202212653, 'learning_rate': 8.617283950617284e-06, 'epoch': 0.45}\n",
      "{'loss': 0.0, 'grad_norm': 0.00011322971076026023, 'learning_rate': 8.613756613756614e-06, 'epoch': 0.45}\n",
      "{'loss': 0.0, 'grad_norm': 1.3006073640228564e-05, 'learning_rate': 8.610229276895943e-06, 'epoch': 0.45}\n",
      "{'loss': 0.0002, 'grad_norm': 0.025902522530526252, 'learning_rate': 8.606701940035274e-06, 'epoch': 0.45}\n",
      "{'loss': 0.0003, 'grad_norm': 0.03434784260784618, 'learning_rate': 8.603174603174604e-06, 'epoch': 0.45}\n",
      "{'loss': 0.0818, 'grad_norm': 7.903585957651769, 'learning_rate': 8.599647266313935e-06, 'epoch': 0.45}\n",
      "{'loss': 0.0023, 'grad_norm': 0.29358080191328023, 'learning_rate': 8.596119929453264e-06, 'epoch': 0.45}\n",
      "{'loss': 0.0, 'grad_norm': 0.0003665959534066689, 'learning_rate': 8.592592592592593e-06, 'epoch': 0.45}\n",
      "{'loss': 0.0, 'grad_norm': 0.0005695044617251786, 'learning_rate': 8.589065255731923e-06, 'epoch': 0.45}\n",
      "{'loss': 0.0001, 'grad_norm': 0.010231531160191439, 'learning_rate': 8.585537918871252e-06, 'epoch': 0.46}\n",
      "{'loss': 0.001, 'grad_norm': 0.10889603271335974, 'learning_rate': 8.582010582010583e-06, 'epoch': 0.46}\n",
      "{'loss': 0.0, 'grad_norm': 0.003205144022088216, 'learning_rate': 8.578483245149911e-06, 'epoch': 0.46}\n",
      "{'loss': 0.0186, 'grad_norm': 2.136501266318423, 'learning_rate': 8.574955908289242e-06, 'epoch': 0.46}\n",
      "{'loss': 0.0008, 'grad_norm': 0.10194565409476601, 'learning_rate': 8.571428571428571e-06, 'epoch': 0.46}\n",
      "{'loss': 0.0469, 'grad_norm': 5.710411120577539, 'learning_rate': 8.567901234567903e-06, 'epoch': 0.46}\n",
      "{'loss': 0.0, 'grad_norm': 0.0016694484314357814, 'learning_rate': 8.564373897707232e-06, 'epoch': 0.46}\n",
      "{'loss': 0.4954, 'grad_norm': 30.429634346872703, 'learning_rate': 8.560846560846563e-06, 'epoch': 0.46}\n",
      "{'loss': 0.0045, 'grad_norm': 0.4825823281518718, 'learning_rate': 8.557319223985891e-06, 'epoch': 0.46}\n",
      "{'loss': 1.1084, 'grad_norm': 36.86766843164804, 'learning_rate': 8.553791887125222e-06, 'epoch': 0.46}\n",
      "{'loss': 0.0185, 'grad_norm': 2.4677376904934674, 'learning_rate': 8.550264550264551e-06, 'epoch': 0.46}\n",
      "{'loss': 0.324, 'grad_norm': 14.182768583129924, 'learning_rate': 8.546737213403881e-06, 'epoch': 0.46}\n",
      "{'loss': 0.0001, 'grad_norm': 0.010444752920372944, 'learning_rate': 8.54320987654321e-06, 'epoch': 0.46}\n",
      "{'loss': 0.0018, 'grad_norm': 0.22388604664612233, 'learning_rate': 8.53968253968254e-06, 'epoch': 0.46}\n",
      "{'loss': 0.0, 'grad_norm': 0.00031651303344376657, 'learning_rate': 8.53615520282187e-06, 'epoch': 0.46}\n",
      "{'loss': 0.0002, 'grad_norm': 0.021984413101060585, 'learning_rate': 8.5326278659612e-06, 'epoch': 0.46}\n",
      "{'loss': 0.0004, 'grad_norm': 0.040202866989868996, 'learning_rate': 8.529100529100531e-06, 'epoch': 0.47}\n",
      "{'loss': 0.0006, 'grad_norm': 0.057329185439642126, 'learning_rate': 8.52557319223986e-06, 'epoch': 0.47}\n",
      "{'loss': 0.0, 'grad_norm': 0.00018165353314853185, 'learning_rate': 8.52204585537919e-06, 'epoch': 0.47}\n",
      "{'loss': 0.0, 'grad_norm': 0.0003078867987902972, 'learning_rate': 8.518518518518519e-06, 'epoch': 0.47}\n",
      "{'loss': 0.0, 'grad_norm': 0.0015058773387693343, 'learning_rate': 8.51499118165785e-06, 'epoch': 0.47}\n",
      "{'loss': 0.0, 'grad_norm': 4.881086829039972e-05, 'learning_rate': 8.511463844797179e-06, 'epoch': 0.47}\n",
      "{'loss': 0.0001, 'grad_norm': 0.007213908732729558, 'learning_rate': 8.507936507936509e-06, 'epoch': 0.47}\n",
      "{'loss': 0.0, 'grad_norm': 0.002654786242672419, 'learning_rate': 8.504409171075838e-06, 'epoch': 0.47}\n",
      "{'loss': 0.0, 'grad_norm': 0.0012058408490961978, 'learning_rate': 8.500881834215169e-06, 'epoch': 0.47}\n",
      "{'loss': 0.0004, 'grad_norm': 0.05370793550945191, 'learning_rate': 8.497354497354499e-06, 'epoch': 0.47}\n",
      "{'loss': 0.0491, 'grad_norm': 10.742294388672756, 'learning_rate': 8.493827160493828e-06, 'epoch': 0.47}\n",
      "{'loss': 3.0234, 'grad_norm': 34.14742673950246, 'learning_rate': 8.490299823633159e-06, 'epoch': 0.47}\n",
      "{'loss': 0.0016, 'grad_norm': 0.17969612432598073, 'learning_rate': 8.486772486772487e-06, 'epoch': 0.47}\n",
      "{'loss': 0.0006, 'grad_norm': 0.07822615512066358, 'learning_rate': 8.483245149911818e-06, 'epoch': 0.47}\n",
      "{'loss': 0.0339, 'grad_norm': 3.888682543109111, 'learning_rate': 8.479717813051147e-06, 'epoch': 0.47}\n",
      "{'loss': 0.8384, 'grad_norm': 28.871385224225474, 'learning_rate': 8.476190476190477e-06, 'epoch': 0.47}\n",
      "{'loss': 0.0004, 'grad_norm': 0.04593262008289041, 'learning_rate': 8.472663139329806e-06, 'epoch': 0.48}\n",
      "{'loss': 0.0413, 'grad_norm': 3.5990813142748936, 'learning_rate': 8.469135802469137e-06, 'epoch': 0.48}\n",
      "{'loss': 0.0, 'grad_norm': 0.00024829324805744656, 'learning_rate': 8.465608465608466e-06, 'epoch': 0.48}\n",
      "{'loss': 0.0014, 'grad_norm': 0.12134661345722246, 'learning_rate': 8.462081128747796e-06, 'epoch': 0.48}\n",
      "{'loss': 0.0001, 'grad_norm': 0.008258756148378845, 'learning_rate': 8.458553791887127e-06, 'epoch': 0.48}\n",
      "{'loss': 0.0, 'grad_norm': 1.5261884668456423e-05, 'learning_rate': 8.455026455026456e-06, 'epoch': 0.48}\n",
      "{'loss': 0.0, 'grad_norm': 0.0006366080998321464, 'learning_rate': 8.451499118165786e-06, 'epoch': 0.48}\n",
      "{'loss': 0.0, 'grad_norm': 0.0018959590479002033, 'learning_rate': 8.447971781305115e-06, 'epoch': 0.48}\n",
      "{'loss': 0.0027, 'grad_norm': 0.25621303838399806, 'learning_rate': 8.444444444444446e-06, 'epoch': 0.48}\n",
      "{'loss': 0.1127, 'grad_norm': 13.165113204544674, 'learning_rate': 8.440917107583775e-06, 'epoch': 0.48}\n",
      "{'loss': 0.1201, 'grad_norm': 12.834868170042995, 'learning_rate': 8.437389770723105e-06, 'epoch': 0.48}\n",
      "{'loss': 0.0, 'grad_norm': 0.0026038684905543875, 'learning_rate': 8.433862433862434e-06, 'epoch': 0.48}\n",
      "{'loss': 0.0, 'grad_norm': 0.004021736295042177, 'learning_rate': 8.430335097001765e-06, 'epoch': 0.48}\n",
      "{'loss': 0.0, 'grad_norm': 0.00032750641938792416, 'learning_rate': 8.426807760141095e-06, 'epoch': 0.48}\n",
      "{'loss': 0.0, 'grad_norm': 8.047626740114877e-05, 'learning_rate': 8.423280423280424e-06, 'epoch': 0.48}\n",
      "{'loss': 0.0085, 'grad_norm': 0.8704230288000193, 'learning_rate': 8.419753086419754e-06, 'epoch': 0.49}\n",
      "{'loss': 0.0, 'grad_norm': 0.004003378932235418, 'learning_rate': 8.416225749559083e-06, 'epoch': 0.49}\n",
      "{'loss': 0.0008, 'grad_norm': 0.10374362146779936, 'learning_rate': 8.412698412698414e-06, 'epoch': 0.49}\n",
      "{'loss': 0.0016, 'grad_norm': 0.1916532091067135, 'learning_rate': 8.409171075837743e-06, 'epoch': 0.49}\n",
      "{'loss': 0.1104, 'grad_norm': 17.01579627869489, 'learning_rate': 8.405643738977073e-06, 'epoch': 0.49}\n",
      "{'loss': 0.0, 'grad_norm': 8.865056815613236e-06, 'learning_rate': 8.402116402116402e-06, 'epoch': 0.49}\n",
      "{'loss': 0.0021, 'grad_norm': 0.17410100991150906, 'learning_rate': 8.398589065255733e-06, 'epoch': 0.49}\n",
      "{'loss': 0.0, 'grad_norm': 0.00011596584931168029, 'learning_rate': 8.395061728395062e-06, 'epoch': 0.49}\n",
      "{'loss': 0.0069, 'grad_norm': 0.7897315635207697, 'learning_rate': 8.391534391534392e-06, 'epoch': 0.49}\n",
      "{'loss': 0.5664, 'grad_norm': 25.892507333882445, 'learning_rate': 8.388007054673723e-06, 'epoch': 0.49}\n",
      "{'loss': 0.0011, 'grad_norm': 0.12514837221172145, 'learning_rate': 8.384479717813052e-06, 'epoch': 0.49}\n",
      "{'loss': 0.0, 'grad_norm': 0.0005045917303995932, 'learning_rate': 8.380952380952382e-06, 'epoch': 0.49}\n",
      "{'loss': 0.0, 'grad_norm': 0.002328967479897718, 'learning_rate': 8.377425044091711e-06, 'epoch': 0.49}\n",
      "{'loss': 0.301, 'grad_norm': 19.529403818995362, 'learning_rate': 8.373897707231042e-06, 'epoch': 0.49}\n",
      "{'loss': 0.0, 'grad_norm': 0.00017831591952772023, 'learning_rate': 8.37037037037037e-06, 'epoch': 0.49}\n",
      "{'loss': 0.314, 'grad_norm': 16.26918751512637, 'learning_rate': 8.366843033509701e-06, 'epoch': 0.49}\n",
      "{'loss': 0.0006, 'grad_norm': 0.06507235067516395, 'learning_rate': 8.36331569664903e-06, 'epoch': 0.5}\n",
      "{'loss': 0.0006, 'grad_norm': 0.050328436631733385, 'learning_rate': 8.35978835978836e-06, 'epoch': 0.5}\n",
      "{'loss': 0.0009, 'grad_norm': 0.10099979553933741, 'learning_rate': 8.356261022927691e-06, 'epoch': 0.5}\n",
      "{'loss': 0.0001, 'grad_norm': 0.008144224107334473, 'learning_rate': 8.35273368606702e-06, 'epoch': 0.5}\n",
      "{'loss': 0.0, 'grad_norm': 9.711874676698687e-06, 'learning_rate': 8.34920634920635e-06, 'epoch': 0.5}\n",
      "{'loss': 0.0848, 'grad_norm': 5.8385652603457645, 'learning_rate': 8.34567901234568e-06, 'epoch': 0.5}\n",
      "{'loss': 0.0, 'grad_norm': 0.0016051862965025753, 'learning_rate': 8.34215167548501e-06, 'epoch': 0.5}\n",
      "{'loss': 0.0002, 'grad_norm': 0.020302463221654438, 'learning_rate': 8.338624338624339e-06, 'epoch': 0.5}\n",
      "{'loss': 0.0001, 'grad_norm': 0.006851699126722101, 'learning_rate': 8.33509700176367e-06, 'epoch': 0.5}\n",
      "{'loss': 0.003, 'grad_norm': 0.24614322255902873, 'learning_rate': 8.331569664902998e-06, 'epoch': 0.5}\n",
      "{'loss': 0.0, 'grad_norm': 0.00042186707325585205, 'learning_rate': 8.328042328042329e-06, 'epoch': 0.5}\n",
      "{'loss': 0.0, 'grad_norm': 0.0006511284727551726, 'learning_rate': 8.324514991181658e-06, 'epoch': 0.5}\n",
      "{'loss': 0.0014, 'grad_norm': 0.14423411988988016, 'learning_rate': 8.320987654320988e-06, 'epoch': 0.5}\n",
      "{'loss': 0.0, 'grad_norm': 6.763410704958312e-05, 'learning_rate': 8.317460317460319e-06, 'epoch': 0.5}\n",
      "{'loss': 0.001, 'grad_norm': 0.07120462538197535, 'learning_rate': 8.313932980599648e-06, 'epoch': 0.5}\n",
      "{'loss': 0.09, 'grad_norm': 7.540055694779124, 'learning_rate': 8.310405643738978e-06, 'epoch': 0.5}\n",
      "{'loss': 0.0002, 'grad_norm': 0.02224578381206101, 'learning_rate': 8.306878306878307e-06, 'epoch': 0.51}\n",
      "{'loss': 0.0002, 'grad_norm': 0.016850629721899414, 'learning_rate': 8.303350970017638e-06, 'epoch': 0.51}\n",
      "{'loss': 0.0, 'grad_norm': 4.9959321868405e-06, 'learning_rate': 8.299823633156966e-06, 'epoch': 0.51}\n",
      "{'loss': 0.0, 'grad_norm': 0.0028996304096124215, 'learning_rate': 8.296296296296297e-06, 'epoch': 0.51}\n",
      "{'loss': 0.0, 'grad_norm': 0.001674502402438329, 'learning_rate': 8.292768959435626e-06, 'epoch': 0.51}\n",
      "{'loss': 0.0, 'grad_norm': 0.0005044578535445088, 'learning_rate': 8.289241622574956e-06, 'epoch': 0.51}\n",
      "{'loss': 0.0125, 'grad_norm': 1.199607264306644, 'learning_rate': 8.285714285714287e-06, 'epoch': 0.51}\n",
      "{'loss': 0.0002, 'grad_norm': 0.019012419790912526, 'learning_rate': 8.282186948853616e-06, 'epoch': 0.51}\n",
      "{'loss': 0.0, 'grad_norm': 0.003598468350873431, 'learning_rate': 8.278659611992946e-06, 'epoch': 0.51}\n",
      "{'loss': 0.0002, 'grad_norm': 0.017287017767825843, 'learning_rate': 8.275132275132275e-06, 'epoch': 0.51}\n",
      "{'loss': 0.0021, 'grad_norm': 0.2663981321781637, 'learning_rate': 8.271604938271606e-06, 'epoch': 0.51}\n",
      "{'loss': 0.0007, 'grad_norm': 0.06172016646172188, 'learning_rate': 8.268077601410935e-06, 'epoch': 0.51}\n",
      "{'loss': 0.0017, 'grad_norm': 0.16455978006722247, 'learning_rate': 8.264550264550265e-06, 'epoch': 0.51}\n",
      "{'loss': 0.0002, 'grad_norm': 0.032395869979947554, 'learning_rate': 8.261022927689594e-06, 'epoch': 0.51}\n",
      "{'loss': 0.0, 'grad_norm': 0.0037173274347592935, 'learning_rate': 8.257495590828925e-06, 'epoch': 0.51}\n",
      "{'loss': 0.0, 'grad_norm': 0.0007528026521051833, 'learning_rate': 8.253968253968254e-06, 'epoch': 0.51}\n",
      "{'loss': 0.0, 'grad_norm': 1.7283513145550473e-05, 'learning_rate': 8.250440917107586e-06, 'epoch': 0.52}\n",
      "{'loss': 0.0002, 'grad_norm': 0.019656667513167636, 'learning_rate': 8.246913580246915e-06, 'epoch': 0.52}\n",
      "{'loss': 0.0, 'grad_norm': 0.0026641396855203413, 'learning_rate': 8.243386243386245e-06, 'epoch': 0.52}\n",
      "{'loss': 0.0002, 'grad_norm': 0.016954450213532965, 'learning_rate': 8.239858906525574e-06, 'epoch': 0.52}\n",
      "{'loss': 0.0, 'grad_norm': 0.0046221999821033114, 'learning_rate': 8.236331569664903e-06, 'epoch': 0.52}\n",
      "{'loss': 0.0046, 'grad_norm': 0.5911409375048218, 'learning_rate': 8.232804232804234e-06, 'epoch': 0.52}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 26%|██▌       | 818/3150 [03:29<10:50,  3.58it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.0002, 'grad_norm': 0.01366528763197738, 'learning_rate': 8.229276895943562e-06, 'epoch': 0.52}\n",
      "{'loss': 0.0, 'grad_norm': 9.54747062331347e-05, 'learning_rate': 8.225749559082893e-06, 'epoch': 0.52}\n",
      "{'loss': 0.0, 'grad_norm': 0.00011539470773831022, 'learning_rate': 8.222222222222222e-06, 'epoch': 0.52}\n",
      "{'loss': 0.2018, 'grad_norm': 16.709705680113448, 'learning_rate': 8.218694885361552e-06, 'epoch': 0.52}\n",
      "{'loss': 0.0, 'grad_norm': 0.0007446771029235906, 'learning_rate': 8.215167548500883e-06, 'epoch': 0.52}\n",
      "{'loss': 0.543, 'grad_norm': 22.15912234003999, 'learning_rate': 8.211640211640213e-06, 'epoch': 0.52}\n",
      "{'loss': 0.0003, 'grad_norm': 0.030122672840349505, 'learning_rate': 8.208112874779542e-06, 'epoch': 0.52}\n",
      "{'loss': 0.0002, 'grad_norm': 0.013163602206137692, 'learning_rate': 8.204585537918873e-06, 'epoch': 0.52}\n",
      "{'loss': 0.0027, 'grad_norm': 0.18348203466131782, 'learning_rate': 8.201058201058202e-06, 'epoch': 0.52}\n",
      "{'loss': 0.0001, 'grad_norm': 0.011142931175322368, 'learning_rate': 8.197530864197532e-06, 'epoch': 0.53}\n",
      "{'loss': 0.0001, 'grad_norm': 0.004195917945753851, 'learning_rate': 8.194003527336861e-06, 'epoch': 0.53}\n",
      "{'loss': 0.0432, 'grad_norm': 5.048397059088255, 'learning_rate': 8.190476190476192e-06, 'epoch': 0.53}\n",
      "{'loss': 0.0001, 'grad_norm': 0.007775929378213971, 'learning_rate': 8.18694885361552e-06, 'epoch': 0.53}\n",
      "{'loss': 0.0, 'grad_norm': 0.0014470615665657422, 'learning_rate': 8.18342151675485e-06, 'epoch': 0.53}\n",
      "{'loss': 0.0, 'grad_norm': 0.0011231901538875367, 'learning_rate': 8.179894179894182e-06, 'epoch': 0.53}\n",
      "{'loss': 0.261, 'grad_norm': 18.480028249215337, 'learning_rate': 8.17636684303351e-06, 'epoch': 0.53}\n",
      "{'loss': 0.1005, 'grad_norm': 8.747367680920634, 'learning_rate': 8.172839506172841e-06, 'epoch': 0.53}\n",
      "{'loss': 0.1167, 'grad_norm': 9.290727599264251, 'learning_rate': 8.16931216931217e-06, 'epoch': 0.53}\n",
      "{'loss': 0.0001, 'grad_norm': 0.012584449809784877, 'learning_rate': 8.1657848324515e-06, 'epoch': 0.53}\n",
      "{'loss': 0.0, 'grad_norm': 0.000601819547446242, 'learning_rate': 8.16225749559083e-06, 'epoch': 0.53}\n",
      "{'loss': 0.0001, 'grad_norm': 0.009366912818183, 'learning_rate': 8.15873015873016e-06, 'epoch': 0.53}\n",
      "{'loss': 0.4741, 'grad_norm': 17.80390331889813, 'learning_rate': 8.155202821869489e-06, 'epoch': 0.53}\n",
      "{'loss': 0.0043, 'grad_norm': 0.5497521058545757, 'learning_rate': 8.15167548500882e-06, 'epoch': 0.53}\n",
      "{'loss': 0.0, 'grad_norm': 0.0006521370960876553, 'learning_rate': 8.148148148148148e-06, 'epoch': 0.53}\n",
      "{'loss': 0.6455, 'grad_norm': 33.86675587239369, 'learning_rate': 8.144620811287479e-06, 'epoch': 0.53}\n",
      "{'loss': 0.0, 'grad_norm': 0.0015057286054526524, 'learning_rate': 8.14109347442681e-06, 'epoch': 0.54}\n",
      "{'loss': 0.0003, 'grad_norm': 0.02981323927249134, 'learning_rate': 8.137566137566138e-06, 'epoch': 0.54}\n",
      "{'loss': 0.0, 'grad_norm': 2.586827527538966e-05, 'learning_rate': 8.134038800705469e-06, 'epoch': 0.54}\n",
      "{'loss': 0.0321, 'grad_norm': 3.597969033295142, 'learning_rate': 8.130511463844798e-06, 'epoch': 0.54}\n",
      "{'loss': 0.0033, 'grad_norm': 0.30835846490293184, 'learning_rate': 8.126984126984128e-06, 'epoch': 0.54}\n",
      "{'loss': 0.0022, 'grad_norm': 0.2603916569462428, 'learning_rate': 8.123456790123457e-06, 'epoch': 0.54}\n",
      "{'loss': 0.174, 'grad_norm': 15.268309561161928, 'learning_rate': 8.119929453262788e-06, 'epoch': 0.54}\n",
      "{'loss': 0.0, 'grad_norm': 0.0035394030083839563, 'learning_rate': 8.116402116402117e-06, 'epoch': 0.54}\n",
      "{'loss': 0.0001, 'grad_norm': 0.007676202238346145, 'learning_rate': 8.112874779541447e-06, 'epoch': 0.54}\n",
      "{'loss': 0.0, 'grad_norm': 0.0017536123939363702, 'learning_rate': 8.109347442680778e-06, 'epoch': 0.54}\n",
      "{'loss': 0.0001, 'grad_norm': 0.005869237167366512, 'learning_rate': 8.105820105820107e-06, 'epoch': 0.54}\n",
      "{'loss': 0.0401, 'grad_norm': 3.7321862865536417, 'learning_rate': 8.102292768959437e-06, 'epoch': 0.54}\n",
      "{'loss': 0.2295, 'grad_norm': 7.993597090916222, 'learning_rate': 8.098765432098766e-06, 'epoch': 0.54}\n",
      "{'loss': 0.0274, 'grad_norm': 1.8120517834217658, 'learning_rate': 8.095238095238097e-06, 'epoch': 0.54}\n",
      "{'loss': 0.0001, 'grad_norm': 0.0075110602096223455, 'learning_rate': 8.091710758377425e-06, 'epoch': 0.54}\n",
      "{'loss': 0.0, 'grad_norm': 0.0020464167610126493, 'learning_rate': 8.088183421516756e-06, 'epoch': 0.54}\n",
      "{'loss': 0.0007, 'grad_norm': 0.07177925756069908, 'learning_rate': 8.084656084656085e-06, 'epoch': 0.55}\n",
      "{'loss': 0.0, 'grad_norm': 0.00066726801114806, 'learning_rate': 8.081128747795415e-06, 'epoch': 0.55}\n",
      "{'loss': 0.0003, 'grad_norm': 0.022370325409600468, 'learning_rate': 8.077601410934744e-06, 'epoch': 0.55}\n",
      "{'loss': 0.2898, 'grad_norm': 7.4943498309344845, 'learning_rate': 8.074074074074075e-06, 'epoch': 0.55}\n",
      "{'loss': 0.0002, 'grad_norm': 0.028934450344406188, 'learning_rate': 8.070546737213405e-06, 'epoch': 0.55}\n",
      "{'loss': 0.3408, 'grad_norm': 24.4835124688534, 'learning_rate': 8.067019400352734e-06, 'epoch': 0.55}\n",
      "{'loss': 0.0059, 'grad_norm': 0.7959642254371811, 'learning_rate': 8.063492063492065e-06, 'epoch': 0.55}\n",
      "{'loss': 0.0, 'grad_norm': 0.0012446552352009376, 'learning_rate': 8.059964726631394e-06, 'epoch': 0.55}\n",
      "{'loss': 0.0061, 'grad_norm': 0.45608700100700533, 'learning_rate': 8.056437389770724e-06, 'epoch': 0.55}\n",
      "{'loss': 0.0, 'grad_norm': 0.004839989327741624, 'learning_rate': 8.052910052910053e-06, 'epoch': 0.55}\n",
      "{'loss': 0.0511, 'grad_norm': 4.670602886811717, 'learning_rate': 8.049382716049384e-06, 'epoch': 0.55}\n",
      "{'loss': 0.0, 'grad_norm': 0.00011118230793122599, 'learning_rate': 8.045855379188713e-06, 'epoch': 0.55}\n",
      "{'loss': 0.0, 'grad_norm': 0.001065913127219991, 'learning_rate': 8.042328042328043e-06, 'epoch': 0.55}\n",
      "{'loss': 0.0, 'grad_norm': 0.005895397721604339, 'learning_rate': 8.038800705467374e-06, 'epoch': 0.55}\n",
      "{'loss': 0.0007, 'grad_norm': 0.05250136457029408, 'learning_rate': 8.035273368606703e-06, 'epoch': 0.55}\n",
      "{'loss': 0.0, 'grad_norm': 0.002524980311282905, 'learning_rate': 8.031746031746033e-06, 'epoch': 0.55}\n",
      "{'loss': 0.7637, 'grad_norm': 26.684650206450616, 'learning_rate': 8.028218694885362e-06, 'epoch': 0.56}\n",
      "{'loss': 0.0124, 'grad_norm': 1.8020111319423544, 'learning_rate': 8.024691358024692e-06, 'epoch': 0.56}\n",
      "{'loss': 0.0002, 'grad_norm': 0.021762275855447218, 'learning_rate': 8.021164021164021e-06, 'epoch': 0.56}\n",
      "{'loss': 0.0, 'grad_norm': 0.00048585361638826054, 'learning_rate': 8.017636684303352e-06, 'epoch': 0.56}\n",
      "{'loss': 0.3779, 'grad_norm': 18.707968025067636, 'learning_rate': 8.01410934744268e-06, 'epoch': 0.56}\n",
      "{'loss': 0.001, 'grad_norm': 0.10774901601105326, 'learning_rate': 8.010582010582011e-06, 'epoch': 0.56}\n",
      "{'loss': 0.0004, 'grad_norm': 0.04167842916271259, 'learning_rate': 8.00705467372134e-06, 'epoch': 0.56}\n",
      "{'loss': 0.113, 'grad_norm': 9.254606182637007, 'learning_rate': 8.00352733686067e-06, 'epoch': 0.56}\n",
      "{'loss': 0.0, 'grad_norm': 0.0006251123955445184, 'learning_rate': 8.000000000000001e-06, 'epoch': 0.56}\n",
      "{'loss': 0.0332, 'grad_norm': 4.2482742845021555, 'learning_rate': 7.99647266313933e-06, 'epoch': 0.56}\n",
      "{'loss': 0.0007, 'grad_norm': 0.08125987566021425, 'learning_rate': 7.99294532627866e-06, 'epoch': 0.56}\n",
      "{'loss': 0.0611, 'grad_norm': 4.411771736887396, 'learning_rate': 7.98941798941799e-06, 'epoch': 0.56}\n",
      "{'loss': 0.0, 'grad_norm': 0.00019737244507706095, 'learning_rate': 7.98589065255732e-06, 'epoch': 0.56}\n",
      "{'loss': 0.0034, 'grad_norm': 0.34659438897203726, 'learning_rate': 7.982363315696649e-06, 'epoch': 0.56}\n",
      "{'loss': 0.0, 'grad_norm': 0.0029172669753065768, 'learning_rate': 7.97883597883598e-06, 'epoch': 0.56}\n",
      "{'loss': 0.0, 'grad_norm': 0.00016008235841995005, 'learning_rate': 7.975308641975308e-06, 'epoch': 0.57}\n",
      "{'loss': 0.0, 'grad_norm': 0.002476588824688619, 'learning_rate': 7.971781305114639e-06, 'epoch': 0.57}\n",
      "{'loss': 0.0, 'grad_norm': 0.001322500744813061, 'learning_rate': 7.968253968253968e-06, 'epoch': 0.57}\n",
      "{'loss': 0.0, 'grad_norm': 0.0005766351850001396, 'learning_rate': 7.964726631393298e-06, 'epoch': 0.57}\n",
      "{'loss': 0.0071, 'grad_norm': 0.7887890785801777, 'learning_rate': 7.961199294532629e-06, 'epoch': 0.57}\n",
      "{'loss': 0.0006, 'grad_norm': 0.08005268867586862, 'learning_rate': 7.957671957671958e-06, 'epoch': 0.57}\n",
      "{'loss': 0.0, 'grad_norm': 0.0006704437062038397, 'learning_rate': 7.954144620811288e-06, 'epoch': 0.57}\n",
      "{'loss': 0.0014, 'grad_norm': 0.1952714876984518, 'learning_rate': 7.950617283950617e-06, 'epoch': 0.57}\n",
      "{'loss': 0.0047, 'grad_norm': 0.441318013647191, 'learning_rate': 7.947089947089948e-06, 'epoch': 0.57}\n",
      "{'loss': 0.0001, 'grad_norm': 0.012120764009431347, 'learning_rate': 7.943562610229277e-06, 'epoch': 0.57}\n",
      "{'loss': 0.0004, 'grad_norm': 0.039469290598816396, 'learning_rate': 7.940035273368607e-06, 'epoch': 0.57}\n",
      "{'loss': 0.0, 'grad_norm': 0.00018203047730550132, 'learning_rate': 7.936507936507936e-06, 'epoch': 0.57}\n",
      "{'loss': 0.0083, 'grad_norm': 0.923199881562237, 'learning_rate': 7.932980599647267e-06, 'epoch': 0.57}\n",
      "{'loss': 0.002, 'grad_norm': 0.2688888551493191, 'learning_rate': 7.929453262786597e-06, 'epoch': 0.57}\n",
      "{'loss': 0.0, 'grad_norm': 0.003875604175901354, 'learning_rate': 7.925925925925926e-06, 'epoch': 0.57}\n",
      "{'loss': 0.1127, 'grad_norm': 5.932678835199247, 'learning_rate': 7.922398589065257e-06, 'epoch': 0.57}\n",
      "{'loss': 0.0038, 'grad_norm': 0.3595546190607289, 'learning_rate': 7.918871252204586e-06, 'epoch': 0.58}\n",
      "{'loss': 0.0034, 'grad_norm': 0.3642995570094504, 'learning_rate': 7.915343915343916e-06, 'epoch': 0.58}\n",
      "{'loss': 0.1221, 'grad_norm': 10.27193727163966, 'learning_rate': 7.911816578483245e-06, 'epoch': 0.58}\n",
      "{'loss': 0.0001, 'grad_norm': 0.005650439764808645, 'learning_rate': 7.908289241622576e-06, 'epoch': 0.58}\n",
      "{'loss': 0.0001, 'grad_norm': 0.00874277474537252, 'learning_rate': 7.904761904761904e-06, 'epoch': 0.58}\n",
      "{'loss': 0.0, 'grad_norm': 0.0021213388168119683, 'learning_rate': 7.901234567901235e-06, 'epoch': 0.58}\n",
      "{'loss': 0.0, 'grad_norm': 0.00022303111898842512, 'learning_rate': 7.897707231040564e-06, 'epoch': 0.58}\n",
      "{'loss': 0.0, 'grad_norm': 0.00019122829166098174, 'learning_rate': 7.894179894179896e-06, 'epoch': 0.58}\n",
      "{'loss': 0.4045, 'grad_norm': 22.06884743478031, 'learning_rate': 7.890652557319225e-06, 'epoch': 0.58}\n",
      "{'loss': 0.0202, 'grad_norm': 2.0428089786876753, 'learning_rate': 7.887125220458554e-06, 'epoch': 0.58}\n",
      "{'loss': 0.0, 'grad_norm': 0.003809598256685286, 'learning_rate': 7.883597883597884e-06, 'epoch': 0.58}\n",
      "{'loss': 0.0005, 'grad_norm': 0.05674425871675961, 'learning_rate': 7.880070546737213e-06, 'epoch': 0.58}\n",
      "{'loss': 0.0, 'grad_norm': 0.0009564351115387099, 'learning_rate': 7.876543209876544e-06, 'epoch': 0.58}\n",
      "{'loss': 0.0, 'grad_norm': 0.0002080755976389167, 'learning_rate': 7.873015873015873e-06, 'epoch': 0.58}\n",
      "{'loss': 0.0001, 'grad_norm': 0.006425745445897408, 'learning_rate': 7.869488536155203e-06, 'epoch': 0.58}\n",
      "{'loss': 0.0909, 'grad_norm': 8.544071163009717, 'learning_rate': 7.865961199294532e-06, 'epoch': 0.58}\n",
      "{'loss': 0.0, 'grad_norm': 0.00023848917035292888, 'learning_rate': 7.862433862433863e-06, 'epoch': 0.59}\n",
      "{'loss': 0.0006, 'grad_norm': 0.06071769743493956, 'learning_rate': 7.858906525573193e-06, 'epoch': 0.59}\n",
      "{'loss': 0.0001, 'grad_norm': 0.008389333275216083, 'learning_rate': 7.855379188712524e-06, 'epoch': 0.59}\n",
      "{'loss': 0.0, 'grad_norm': 0.002705258534927597, 'learning_rate': 7.851851851851853e-06, 'epoch': 0.59}\n",
      "{'loss': 0.0, 'grad_norm': 3.604109606746895e-05, 'learning_rate': 7.848324514991183e-06, 'epoch': 0.59}\n",
      "{'loss': 0.0, 'grad_norm': 0.0002807183749923046, 'learning_rate': 7.844797178130512e-06, 'epoch': 0.59}\n",
      "{'loss': 0.5972, 'grad_norm': 26.764690037669034, 'learning_rate': 7.841269841269843e-06, 'epoch': 0.59}\n",
      "{'loss': 0.0334, 'grad_norm': 3.342835631085908, 'learning_rate': 7.837742504409172e-06, 'epoch': 0.59}\n",
      "{'loss': 0.0, 'grad_norm': 0.0038163927601852347, 'learning_rate': 7.8342151675485e-06, 'epoch': 0.59}\n",
      "{'loss': 0.0001, 'grad_norm': 0.006061960858466976, 'learning_rate': 7.830687830687831e-06, 'epoch': 0.59}\n",
      "{'loss': 0.0001, 'grad_norm': 0.006897625639057729, 'learning_rate': 7.82716049382716e-06, 'epoch': 0.59}\n",
      "{'loss': 0.0411, 'grad_norm': 4.510786057544028, 'learning_rate': 7.823633156966492e-06, 'epoch': 0.59}\n",
      "{'loss': 0.002, 'grad_norm': 0.16688981738724135, 'learning_rate': 7.820105820105821e-06, 'epoch': 0.59}\n",
      "{'loss': 0.0, 'grad_norm': 0.0002271676819097237, 'learning_rate': 7.816578483245151e-06, 'epoch': 0.59}\n",
      "{'loss': 1.0898, 'grad_norm': 30.792457112725877, 'learning_rate': 7.81305114638448e-06, 'epoch': 0.59}\n",
      "{'loss': 0.0, 'grad_norm': 3.939256698959672e-06, 'learning_rate': 7.809523809523811e-06, 'epoch': 0.59}\n",
      "{'loss': 0.0006, 'grad_norm': 0.06979931874221132, 'learning_rate': 7.80599647266314e-06, 'epoch': 0.6}\n",
      "{'loss': 0.0001, 'grad_norm': 0.009009910771464772, 'learning_rate': 7.80246913580247e-06, 'epoch': 0.6}\n",
      "{'loss': 0.0049, 'grad_norm': 0.5208071479572722, 'learning_rate': 7.7989417989418e-06, 'epoch': 0.6}\n",
      "{'loss': 0.01, 'grad_norm': 1.2919247379703322, 'learning_rate': 7.79541446208113e-06, 'epoch': 0.6}\n",
      "{'loss': 0.0, 'grad_norm': 2.3390956283098967e-05, 'learning_rate': 7.791887125220459e-06, 'epoch': 0.6}\n",
      "{'loss': 0.0001, 'grad_norm': 0.009077033868249271, 'learning_rate': 7.78835978835979e-06, 'epoch': 0.6}\n",
      "{'loss': 0.0001, 'grad_norm': 0.009741458943443844, 'learning_rate': 7.78483245149912e-06, 'epoch': 0.6}\n",
      "{'loss': 0.0, 'grad_norm': 0.0010361953420791729, 'learning_rate': 7.781305114638449e-06, 'epoch': 0.6}\n",
      "{'loss': 0.0005, 'grad_norm': 0.04700746297492389, 'learning_rate': 7.77777777777778e-06, 'epoch': 0.6}\n",
      "{'loss': 0.0222, 'grad_norm': 2.1423778179498942, 'learning_rate': 7.774250440917108e-06, 'epoch': 0.6}\n",
      "{'loss': 0.0, 'grad_norm': 0.0017619233008528002, 'learning_rate': 7.770723104056439e-06, 'epoch': 0.6}\n",
      "{'loss': 0.0036, 'grad_norm': 0.38719431217657285, 'learning_rate': 7.767195767195767e-06, 'epoch': 0.6}\n",
      "{'loss': 0.0, 'grad_norm': 0.0004991794734748026, 'learning_rate': 7.763668430335098e-06, 'epoch': 0.6}\n",
      "{'loss': 0.0006, 'grad_norm': 0.05977621607965379, 'learning_rate': 7.760141093474427e-06, 'epoch': 0.6}\n",
      "{'loss': 0.0001, 'grad_norm': 0.008572443425155925, 'learning_rate': 7.756613756613757e-06, 'epoch': 0.6}\n",
      "{'loss': 0.0881, 'grad_norm': 4.718899149779641, 'learning_rate': 7.753086419753088e-06, 'epoch': 0.61}\n",
      "{'loss': 0.0013, 'grad_norm': 0.1542819595370822, 'learning_rate': 7.749559082892417e-06, 'epoch': 0.61}\n",
      "{'loss': 0.0004, 'grad_norm': 0.03851612309574846, 'learning_rate': 7.746031746031747e-06, 'epoch': 0.61}\n",
      "{'loss': 0.0001, 'grad_norm': 0.008186494091683937, 'learning_rate': 7.742504409171076e-06, 'epoch': 0.61}\n",
      "{'loss': 0.0004, 'grad_norm': 0.052558778764618105, 'learning_rate': 7.738977072310407e-06, 'epoch': 0.61}\n",
      "{'loss': 0.0014, 'grad_norm': 0.1367223398554755, 'learning_rate': 7.735449735449736e-06, 'epoch': 0.61}\n",
      "{'loss': 0.0001, 'grad_norm': 0.006748342862931809, 'learning_rate': 7.731922398589066e-06, 'epoch': 0.61}\n",
      "{'loss': 0.0011, 'grad_norm': 0.13083930359822996, 'learning_rate': 7.728395061728395e-06, 'epoch': 0.61}\n",
      "{'loss': 0.0034, 'grad_norm': 0.34909767125660696, 'learning_rate': 7.724867724867726e-06, 'epoch': 0.61}\n",
      "{'loss': 0.0002, 'grad_norm': 0.016420061181984958, 'learning_rate': 7.721340388007055e-06, 'epoch': 0.61}\n",
      "{'loss': 0.0097, 'grad_norm': 1.0500982965144403, 'learning_rate': 7.717813051146385e-06, 'epoch': 0.61}\n",
      "{'loss': 0.0, 'grad_norm': 0.0021884820468119786, 'learning_rate': 7.714285714285716e-06, 'epoch': 0.61}\n",
      "{'loss': 0.0, 'grad_norm': 0.002702503441874129, 'learning_rate': 7.710758377425045e-06, 'epoch': 0.61}\n",
      "{'loss': 0.0, 'grad_norm': 0.003970984421517217, 'learning_rate': 7.707231040564375e-06, 'epoch': 0.61}\n",
      "{'loss': 0.0, 'grad_norm': 0.003042625681359637, 'learning_rate': 7.703703703703704e-06, 'epoch': 0.61}\n",
      "{'loss': 0.0116, 'grad_norm': 1.2510121062780812, 'learning_rate': 7.700176366843035e-06, 'epoch': 0.61}\n",
      "{'loss': 3.2188, 'grad_norm': 32.19235589888246, 'learning_rate': 7.696649029982363e-06, 'epoch': 0.62}\n",
      "{'loss': 0.0001, 'grad_norm': 0.004617785973185586, 'learning_rate': 7.693121693121694e-06, 'epoch': 0.62}\n",
      "{'loss': 0.0001, 'grad_norm': 0.009512931852363361, 'learning_rate': 7.689594356261023e-06, 'epoch': 0.62}\n",
      "{'loss': 0.5762, 'grad_norm': 18.318075223199966, 'learning_rate': 7.686067019400353e-06, 'epoch': 0.62}\n",
      "{'loss': 0.2094, 'grad_norm': 15.391769568005179, 'learning_rate': 7.682539682539684e-06, 'epoch': 0.62}\n",
      "{'loss': 1.2949, 'grad_norm': 30.7653393206645, 'learning_rate': 7.679012345679013e-06, 'epoch': 0.62}\n",
      "{'loss': 0.0, 'grad_norm': 0.003166902130769312, 'learning_rate': 7.675485008818343e-06, 'epoch': 0.62}\n",
      "{'loss': 0.0008, 'grad_norm': 0.09633251432332737, 'learning_rate': 7.671957671957672e-06, 'epoch': 0.62}\n",
      "{'loss': 0.0001, 'grad_norm': 0.005506288228433534, 'learning_rate': 7.668430335097003e-06, 'epoch': 0.62}\n",
      "{'loss': 0.0, 'grad_norm': 0.0015381454848170696, 'learning_rate': 7.664902998236332e-06, 'epoch': 0.62}\n",
      "{'loss': 0.0003, 'grad_norm': 0.0367548391733793, 'learning_rate': 7.661375661375662e-06, 'epoch': 0.62}\n",
      "{'loss': 0.0, 'grad_norm': 0.001258098267606817, 'learning_rate': 7.657848324514991e-06, 'epoch': 0.62}\n",
      "{'loss': 0.0007, 'grad_norm': 0.07834750319446594, 'learning_rate': 7.654320987654322e-06, 'epoch': 0.62}\n",
      "{'loss': 0.0, 'grad_norm': 0.0002310705726260139, 'learning_rate': 7.65079365079365e-06, 'epoch': 0.62}\n",
      "{'loss': 0.0, 'grad_norm': 0.0033385041841621755, 'learning_rate': 7.647266313932981e-06, 'epoch': 0.62}\n",
      "{'loss': 0.0, 'grad_norm': 0.001532137745248932, 'learning_rate': 7.643738977072312e-06, 'epoch': 0.62}\n",
      "{'loss': 0.0, 'grad_norm': 0.0023667449084726214, 'learning_rate': 7.64021164021164e-06, 'epoch': 0.63}\n",
      "{'loss': 0.0516, 'grad_norm': 5.0686444329158755, 'learning_rate': 7.636684303350971e-06, 'epoch': 0.63}\n",
      "{'loss': 0.0, 'grad_norm': 0.00010510793233403064, 'learning_rate': 7.6331569664903e-06, 'epoch': 0.63}\n",
      "{'loss': 0.0, 'grad_norm': 0.00033340568673222875, 'learning_rate': 7.62962962962963e-06, 'epoch': 0.63}\n",
      "{'loss': 0.0, 'grad_norm': 0.0002884817393953668, 'learning_rate': 7.62610229276896e-06, 'epoch': 0.63}\n",
      "{'loss': 0.0002, 'grad_norm': 0.016880072900762766, 'learning_rate': 7.62257495590829e-06, 'epoch': 0.63}\n",
      "{'loss': 0.4253, 'grad_norm': 29.60121197667895, 'learning_rate': 7.61904761904762e-06, 'epoch': 0.63}\n",
      "{'loss': 0.0166, 'grad_norm': 2.725451923658054, 'learning_rate': 7.615520282186949e-06, 'epoch': 0.63}\n",
      "{'loss': 0.0025, 'grad_norm': 0.26824900482789493, 'learning_rate': 7.61199294532628e-06, 'epoch': 0.63}\n",
      "{'loss': 0.001, 'grad_norm': 0.11006989442203186, 'learning_rate': 7.60846560846561e-06, 'epoch': 0.63}\n",
      "{'loss': 0.0002, 'grad_norm': 0.015873227143475617, 'learning_rate': 7.604938271604939e-06, 'epoch': 0.63}\n",
      "{'loss': 0.0043, 'grad_norm': 0.47639718862531993, 'learning_rate': 7.601410934744269e-06, 'epoch': 0.63}\n",
      "{'loss': 0.0, 'grad_norm': 0.0006231376417985171, 'learning_rate': 7.597883597883599e-06, 'epoch': 0.63}\n",
      "{'loss': 0.018, 'grad_norm': 2.2003035639356714, 'learning_rate': 7.5943562610229285e-06, 'epoch': 0.63}\n",
      "{'loss': 0.0087, 'grad_norm': 0.9316108746846773, 'learning_rate': 7.590828924162258e-06, 'epoch': 0.63}\n",
      "{'loss': 0.0, 'grad_norm': 0.0008231630277249667, 'learning_rate': 7.587301587301588e-06, 'epoch': 0.63}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 32%|███▏      | 1000/3150 [04:16<08:47,  4.08it/s]12/23/2024 06:39:23 - INFO - FlagEmbedding.finetune.embedder.encoder_only.base.trainer -   Saving model checkpoint to ./test_encoder_only_base_bge-large-en-v1.5/checkpoint-1000\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.0001, 'grad_norm': 0.0052094104905307205, 'learning_rate': 7.583774250440918e-06, 'epoch': 0.64}\n",
      "{'loss': 0.0004, 'grad_norm': 0.05414591780232195, 'learning_rate': 7.580246913580247e-06, 'epoch': 0.64}\n",
      "{'loss': 0.0, 'grad_norm': 0.005033967507836883, 'learning_rate': 7.576719576719578e-06, 'epoch': 0.64}\n",
      "{'loss': 0.0002, 'grad_norm': 0.01698784361595978, 'learning_rate': 7.573192239858908e-06, 'epoch': 0.64}\n",
      "{'loss': 0.0, 'grad_norm': 0.00047723063982967767, 'learning_rate': 7.569664902998237e-06, 'epoch': 0.64}\n",
      "{'loss': 0.0006, 'grad_norm': 0.0427643550196247, 'learning_rate': 7.566137566137567e-06, 'epoch': 0.64}\n",
      "{'loss': 0.0189, 'grad_norm': 2.0302958668418953, 'learning_rate': 7.562610229276897e-06, 'epoch': 0.64}\n",
      "{'loss': 0.0001, 'grad_norm': 0.00556046268974225, 'learning_rate': 7.5590828924162264e-06, 'epoch': 0.64}\n",
      "{'loss': 0.0, 'grad_norm': 0.0005143339470081945, 'learning_rate': 7.555555555555556e-06, 'epoch': 0.64}\n",
      "{'loss': 0.0001, 'grad_norm': 0.010744205740422917, 'learning_rate': 7.552028218694886e-06, 'epoch': 0.64}\n",
      "{'loss': 0.0, 'grad_norm': 0.00010216109619030751, 'learning_rate': 7.548500881834216e-06, 'epoch': 0.64}\n",
      "{'loss': 0.0, 'grad_norm': 0.00024233421950172856, 'learning_rate': 7.544973544973545e-06, 'epoch': 0.64}\n",
      "{'loss': 0.0, 'grad_norm': 1.0216342232561567e-05, 'learning_rate': 7.541446208112876e-06, 'epoch': 0.64}\n",
      "{'loss': 0.0, 'grad_norm': 0.003740421669930925, 'learning_rate': 7.5379188712522056e-06, 'epoch': 0.64}\n",
      "{'loss': 0.0003, 'grad_norm': 0.021698335384745786, 'learning_rate': 7.534391534391535e-06, 'epoch': 0.64}\n",
      "{'loss': 0.0072, 'grad_norm': 1.054867481840003, 'learning_rate': 7.530864197530865e-06, 'epoch': 0.65}\n",
      "{'loss': 0.0118, 'grad_norm': 1.0080957532165276, 'learning_rate': 7.527336860670195e-06, 'epoch': 0.65}\n",
      "{'loss': 0.0, 'grad_norm': 0.0005697816971277557, 'learning_rate': 7.523809523809524e-06, 'epoch': 0.65}\n",
      "{'loss': 0.0, 'grad_norm': 0.00023787033854209285, 'learning_rate': 7.520282186948854e-06, 'epoch': 0.65}\n",
      "{'loss': 0.0, 'grad_norm': 0.00016250391226935732, 'learning_rate': 7.516754850088184e-06, 'epoch': 0.65}\n",
      "{'loss': 0.0, 'grad_norm': 0.0022453854201322484, 'learning_rate': 7.5132275132275136e-06, 'epoch': 0.65}\n",
      "{'loss': 0.0096, 'grad_norm': 1.0551639752166635, 'learning_rate': 7.509700176366843e-06, 'epoch': 0.65}\n",
      "{'loss': 0.0, 'grad_norm': 6.12781225585108e-05, 'learning_rate': 7.506172839506174e-06, 'epoch': 0.65}\n",
      "{'loss': 0.0, 'grad_norm': 0.0007724140136321605, 'learning_rate': 7.5026455026455035e-06, 'epoch': 0.65}\n",
      "{'loss': 0.0117, 'grad_norm': 0.8927948603024087, 'learning_rate': 7.499118165784833e-06, 'epoch': 0.65}\n",
      "{'loss': 0.0, 'grad_norm': 1.3815692985261067e-05, 'learning_rate': 7.495590828924163e-06, 'epoch': 0.65}\n",
      "{'loss': 0.0003, 'grad_norm': 0.044958211362933, 'learning_rate': 7.492063492063493e-06, 'epoch': 0.65}\n",
      "{'loss': 0.1357, 'grad_norm': 11.84201834306551, 'learning_rate': 7.488536155202822e-06, 'epoch': 0.65}\n",
      "{'loss': 0.0, 'grad_norm': 1.6532686752138374e-06, 'learning_rate': 7.485008818342152e-06, 'epoch': 0.65}\n",
      "{'loss': 0.0013, 'grad_norm': 0.15584331874317311, 'learning_rate': 7.481481481481482e-06, 'epoch': 0.65}\n",
      "{'loss': 0.0035, 'grad_norm': 0.36025654436873966, 'learning_rate': 7.4779541446208115e-06, 'epoch': 0.65}\n",
      "{'loss': 0.0, 'grad_norm': 0.0014822316433413434, 'learning_rate': 7.474426807760141e-06, 'epoch': 0.66}\n",
      "{'loss': 0.0001, 'grad_norm': 0.010285565627017499, 'learning_rate': 7.470899470899472e-06, 'epoch': 0.66}\n",
      "{'loss': 0.0, 'grad_norm': 8.192606657720842e-05, 'learning_rate': 7.4673721340388015e-06, 'epoch': 0.66}\n",
      "{'loss': 0.0004, 'grad_norm': 0.048262416115928485, 'learning_rate': 7.463844797178131e-06, 'epoch': 0.66}\n",
      "{'loss': 0.0028, 'grad_norm': 0.30089557010968654, 'learning_rate': 7.460317460317461e-06, 'epoch': 0.66}\n",
      "{'loss': 0.1298, 'grad_norm': 8.155572738883441, 'learning_rate': 7.456790123456791e-06, 'epoch': 0.66}\n",
      "{'loss': 0.0, 'grad_norm': 0.00016864204437984256, 'learning_rate': 7.45326278659612e-06, 'epoch': 0.66}\n",
      "{'loss': 0.0, 'grad_norm': 0.0011889716772284782, 'learning_rate': 7.44973544973545e-06, 'epoch': 0.66}\n",
      "{'loss': 0.0127, 'grad_norm': 1.510256984644656, 'learning_rate': 7.44620811287478e-06, 'epoch': 0.66}\n",
      "{'loss': 0.0, 'grad_norm': 0.004336441954501604, 'learning_rate': 7.4426807760141095e-06, 'epoch': 0.66}\n",
      "{'loss': 0.0003, 'grad_norm': 0.03405753515090086, 'learning_rate': 7.439153439153439e-06, 'epoch': 0.66}\n",
      "{'loss': 0.0004, 'grad_norm': 0.05569069417696394, 'learning_rate': 7.43562610229277e-06, 'epoch': 0.66}\n",
      "{'loss': 0.0005, 'grad_norm': 0.03878594190505073, 'learning_rate': 7.4320987654320995e-06, 'epoch': 0.66}\n",
      "{'loss': 0.0, 'grad_norm': 2.887554575139712e-05, 'learning_rate': 7.428571428571429e-06, 'epoch': 0.66}\n",
      "{'loss': 0.0346, 'grad_norm': 6.087757156862265, 'learning_rate': 7.425044091710759e-06, 'epoch': 0.66}\n",
      "{'loss': 0.0, 'grad_norm': 0.0025767411754557983, 'learning_rate': 7.421516754850089e-06, 'epoch': 0.66}\n",
      "{'loss': 0.0001, 'grad_norm': 0.009095111907369932, 'learning_rate': 7.417989417989418e-06, 'epoch': 0.67}\n",
      "{'loss': 0.0003, 'grad_norm': 0.028599141324644705, 'learning_rate': 7.414462081128748e-06, 'epoch': 0.67}\n",
      "{'loss': 0.4458, 'grad_norm': 0.028599141324644705, 'learning_rate': 7.414462081128748e-06, 'epoch': 0.67}\n",
      "{'loss': 0.0, 'grad_norm': 0.0010738995827883776, 'learning_rate': 7.410934744268078e-06, 'epoch': 0.67}\n",
      "{'loss': 0.0, 'grad_norm': 0.00422952978825761, 'learning_rate': 7.4074074074074075e-06, 'epoch': 0.67}\n",
      "{'loss': 0.0, 'grad_norm': 0.0007599708799328013, 'learning_rate': 7.403880070546737e-06, 'epoch': 0.67}\n",
      "{'loss': 0.0, 'grad_norm': 0.0007646528391081275, 'learning_rate': 7.400352733686068e-06, 'epoch': 0.67}\n",
      "{'loss': 0.0092, 'grad_norm': 1.1521183116677731, 'learning_rate': 7.3968253968253975e-06, 'epoch': 0.67}\n",
      "{'loss': 0.0006, 'grad_norm': 0.062322360405253165, 'learning_rate': 7.393298059964727e-06, 'epoch': 0.67}\n",
      "{'loss': 0.0001, 'grad_norm': 0.00583621660242817, 'learning_rate': 7.389770723104057e-06, 'epoch': 0.67}\n",
      "{'loss': 0.011, 'grad_norm': 0.7165663761744446, 'learning_rate': 7.386243386243387e-06, 'epoch': 0.67}\n",
      "{'loss': 0.0001, 'grad_norm': 0.01506913531324177, 'learning_rate': 7.382716049382716e-06, 'epoch': 0.67}\n",
      "{'loss': 0.0028, 'grad_norm': 0.281592041883367, 'learning_rate': 7.379188712522046e-06, 'epoch': 0.67}\n",
      "{'loss': 0.0, 'grad_norm': 0.0004990047321497131, 'learning_rate': 7.375661375661376e-06, 'epoch': 0.67}\n",
      "{'loss': 0.0, 'grad_norm': 0.0037314028297612666, 'learning_rate': 7.3721340388007055e-06, 'epoch': 0.67}\n",
      "{'loss': 0.0002, 'grad_norm': 0.023560310263205757, 'learning_rate': 7.368606701940035e-06, 'epoch': 0.67}\n",
      "{'loss': 0.0338, 'grad_norm': 2.2635815053547943, 'learning_rate': 7.3650793650793666e-06, 'epoch': 0.68}\n",
      "{'loss': 0.0, 'grad_norm': 0.004171264404611568, 'learning_rate': 7.3615520282186954e-06, 'epoch': 0.68}\n",
      "{'loss': 0.004, 'grad_norm': 0.623662590560798, 'learning_rate': 7.358024691358025e-06, 'epoch': 0.68}\n",
      "{'loss': 0.0, 'grad_norm': 0.0003950455204952606, 'learning_rate': 7.354497354497355e-06, 'epoch': 0.68}\n",
      "{'loss': 0.0003, 'grad_norm': 0.035785913949495735, 'learning_rate': 7.350970017636685e-06, 'epoch': 0.68}\n",
      "{'loss': 0.0722, 'grad_norm': 5.543554725347259, 'learning_rate': 7.347442680776014e-06, 'epoch': 0.68}\n",
      "{'loss': 0.1495, 'grad_norm': 8.66195193451903, 'learning_rate': 7.343915343915344e-06, 'epoch': 0.68}\n",
      "{'loss': 0.0003, 'grad_norm': 0.033861506603272494, 'learning_rate': 7.340388007054674e-06, 'epoch': 0.68}\n",
      "{'loss': 0.0, 'grad_norm': 0.0016614092578933683, 'learning_rate': 7.3368606701940034e-06, 'epoch': 0.68}\n",
      "{'loss': 0.0006, 'grad_norm': 0.07013870655515837, 'learning_rate': 7.333333333333333e-06, 'epoch': 0.68}\n",
      "{'loss': 0.292, 'grad_norm': 18.878341682149117, 'learning_rate': 7.3298059964726645e-06, 'epoch': 0.68}\n",
      "{'loss': 0.0005, 'grad_norm': 0.056812666359795296, 'learning_rate': 7.326278659611994e-06, 'epoch': 0.68}\n",
      "{'loss': 0.0, 'grad_norm': 6.452788492670502e-05, 'learning_rate': 7.322751322751324e-06, 'epoch': 0.68}\n",
      "{'loss': 0.0, 'grad_norm': 0.0005099433991959939, 'learning_rate': 7.319223985890654e-06, 'epoch': 0.68}\n",
      "{'loss': 0.0, 'grad_norm': 0.0036985517537734803, 'learning_rate': 7.315696649029983e-06, 'epoch': 0.68}\n",
      "{'loss': 0.0001, 'grad_norm': 0.00696819640712324, 'learning_rate': 7.312169312169313e-06, 'epoch': 0.69}\n",
      "{'loss': 0.0015, 'grad_norm': 0.1392208368613793, 'learning_rate': 7.308641975308642e-06, 'epoch': 0.69}\n",
      "{'loss': 0.0001, 'grad_norm': 0.01718100518548384, 'learning_rate': 7.305114638447972e-06, 'epoch': 0.69}\n",
      "{'loss': 0.0001, 'grad_norm': 0.009250585024950238, 'learning_rate': 7.301587301587301e-06, 'epoch': 0.69}\n",
      "{'loss': 0.1979, 'grad_norm': 7.639917758279306, 'learning_rate': 7.298059964726631e-06, 'epoch': 0.69}\n",
      "{'loss': 0.0005, 'grad_norm': 0.05810349124460543, 'learning_rate': 7.2945326278659625e-06, 'epoch': 0.69}\n",
      "{'loss': 0.0, 'grad_norm': 2.821188838800522e-06, 'learning_rate': 7.291005291005292e-06, 'epoch': 0.69}\n",
      "{'loss': 0.0, 'grad_norm': 0.004157038450311585, 'learning_rate': 7.287477954144622e-06, 'epoch': 0.69}\n",
      "{'loss': 0.0, 'grad_norm': 8.496457348868182e-05, 'learning_rate': 7.283950617283952e-06, 'epoch': 0.69}\n",
      "{'loss': 0.0, 'grad_norm': 0.0015336001514569795, 'learning_rate': 7.280423280423281e-06, 'epoch': 0.69}\n",
      "{'loss': 0.0001, 'grad_norm': 0.00947647083977476, 'learning_rate': 7.276895943562611e-06, 'epoch': 0.69}\n",
      "{'loss': 0.0, 'grad_norm': 0.00255975566754674, 'learning_rate': 7.273368606701941e-06, 'epoch': 0.69}\n",
      "{'loss': 1.416, 'grad_norm': 28.52389113923022, 'learning_rate': 7.2698412698412705e-06, 'epoch': 0.69}\n",
      "{'loss': 0.0, 'grad_norm': 0.005028682801166208, 'learning_rate': 7.2663139329806e-06, 'epoch': 0.69}\n",
      "{'loss': 0.0, 'grad_norm': 0.0006450510601520889, 'learning_rate': 7.26278659611993e-06, 'epoch': 0.69}\n",
      "{'loss': 0.0, 'grad_norm': 1.7259415088122994e-05, 'learning_rate': 7.2592592592592605e-06, 'epoch': 0.69}\n",
      "{'loss': 0.0011, 'grad_norm': 0.09513226865393849, 'learning_rate': 7.25573192239859e-06, 'epoch': 0.7}\n",
      "{'loss': 0.0, 'grad_norm': 0.0017292162110746924, 'learning_rate': 7.25220458553792e-06, 'epoch': 0.7}\n",
      "{'loss': 0.0004, 'grad_norm': 0.0510805849042812, 'learning_rate': 7.24867724867725e-06, 'epoch': 0.7}\n",
      "{'loss': 0.0, 'grad_norm': 1.4329058316929913e-05, 'learning_rate': 7.245149911816579e-06, 'epoch': 0.7}\n",
      "{'loss': 0.0, 'grad_norm': 0.00015443637650619684, 'learning_rate': 7.241622574955909e-06, 'epoch': 0.7}\n",
      "{'loss': 0.0001, 'grad_norm': 0.007128459007215735, 'learning_rate': 7.238095238095239e-06, 'epoch': 0.7}\n",
      "{'loss': 0.0, 'grad_norm': 0.0003886365997567944, 'learning_rate': 7.2345679012345685e-06, 'epoch': 0.7}\n",
      "{'loss': 0.0183, 'grad_norm': 2.210138291049672, 'learning_rate': 7.231040564373898e-06, 'epoch': 0.7}\n",
      "{'loss': 0.0, 'grad_norm': 0.00042370063930431706, 'learning_rate': 7.227513227513228e-06, 'epoch': 0.7}\n",
      "{'loss': 0.0096, 'grad_norm': 1.402428032087651, 'learning_rate': 7.2239858906525585e-06, 'epoch': 0.7}\n",
      "{'loss': 0.0003, 'grad_norm': 0.038017194098280395, 'learning_rate': 7.220458553791888e-06, 'epoch': 0.7}\n",
      "{'loss': 0.0001, 'grad_norm': 0.006719318257741503, 'learning_rate': 7.216931216931218e-06, 'epoch': 0.7}\n",
      "{'loss': 0.0, 'grad_norm': 0.0013119092749737868, 'learning_rate': 7.213403880070548e-06, 'epoch': 0.7}\n",
      "{'loss': 0.0, 'grad_norm': 0.00326349357630025, 'learning_rate': 7.209876543209877e-06, 'epoch': 0.7}\n",
      "{'loss': 0.0, 'grad_norm': 8.424545405099231e-05, 'learning_rate': 7.206349206349207e-06, 'epoch': 0.7}\n",
      "{'loss': 0.0, 'grad_norm': 0.0003982439278980188, 'learning_rate': 7.202821869488537e-06, 'epoch': 0.7}\n",
      "{'loss': 0.0, 'grad_norm': 0.004100651849259906, 'learning_rate': 7.1992945326278665e-06, 'epoch': 0.71}\n",
      "{'loss': 0.0005, 'grad_norm': 0.08376795863096015, 'learning_rate': 7.195767195767196e-06, 'epoch': 0.71}\n",
      "{'loss': 0.0, 'grad_norm': 0.0003806433257178503, 'learning_rate': 7.192239858906526e-06, 'epoch': 0.71}\n",
      "{'loss': 0.0, 'grad_norm': 0.0004371985864191575, 'learning_rate': 7.1887125220458564e-06, 'epoch': 0.71}\n",
      "{'loss': 0.0, 'grad_norm': 0.002202685446877771, 'learning_rate': 7.185185185185186e-06, 'epoch': 0.71}\n",
      "{'loss': 0.0, 'grad_norm': 0.0011774378229605727, 'learning_rate': 7.181657848324516e-06, 'epoch': 0.71}\n",
      "{'loss': 0.0, 'grad_norm': 0.0002363319054221631, 'learning_rate': 7.178130511463846e-06, 'epoch': 0.71}\n",
      "{'loss': 0.0, 'grad_norm': 2.1467973049351387e-05, 'learning_rate': 7.174603174603175e-06, 'epoch': 0.71}\n",
      "{'loss': 0.0048, 'grad_norm': 0.5323783727997174, 'learning_rate': 7.171075837742505e-06, 'epoch': 0.71}\n",
      "{'loss': 0.0, 'grad_norm': 0.0017541108016193648, 'learning_rate': 7.167548500881835e-06, 'epoch': 0.71}\n",
      "{'loss': 0.0, 'grad_norm': 4.0251321567609974e-05, 'learning_rate': 7.1640211640211644e-06, 'epoch': 0.71}\n",
      "{'loss': 0.0, 'grad_norm': 0.00016561387862938241, 'learning_rate': 7.160493827160494e-06, 'epoch': 0.71}\n",
      "{'loss': 0.0001, 'grad_norm': 0.009459194746214395, 'learning_rate': 7.156966490299824e-06, 'epoch': 0.71}\n",
      "{'loss': 0.0003, 'grad_norm': 0.0365645321511539, 'learning_rate': 7.1534391534391544e-06, 'epoch': 0.71}\n",
      "{'loss': 1.5635, 'grad_norm': 30.519713987757672, 'learning_rate': 7.149911816578484e-06, 'epoch': 0.71}\n",
      "{'loss': 0.0003, 'grad_norm': 0.03408260351810018, 'learning_rate': 7.146384479717814e-06, 'epoch': 0.71}\n",
      "{'loss': 0.0, 'grad_norm': 1.84518302899523e-05, 'learning_rate': 7.1428571428571436e-06, 'epoch': 0.72}\n",
      "{'loss': 0.001, 'grad_norm': 0.07008096535028749, 'learning_rate': 7.139329805996473e-06, 'epoch': 0.72}\n",
      "{'loss': 0.0478, 'grad_norm': 4.544343538622656, 'learning_rate': 7.135802469135803e-06, 'epoch': 0.72}\n",
      "{'loss': 0.0001, 'grad_norm': 0.015455315057891281, 'learning_rate': 7.132275132275133e-06, 'epoch': 0.72}\n",
      "{'loss': 0.0, 'grad_norm': 0.0011322714949523805, 'learning_rate': 7.128747795414462e-06, 'epoch': 0.72}\n",
      "{'loss': 0.0062, 'grad_norm': 0.48513933603777176, 'learning_rate': 7.125220458553792e-06, 'epoch': 0.72}\n",
      "{'loss': 0.0, 'grad_norm': 0.0009347122476299465, 'learning_rate': 7.121693121693122e-06, 'epoch': 0.72}\n",
      "{'loss': 0.2876, 'grad_norm': 22.927391548170608, 'learning_rate': 7.118165784832452e-06, 'epoch': 0.72}\n",
      "{'loss': 0.0034, 'grad_norm': 0.43529349171255377, 'learning_rate': 7.114638447971782e-06, 'epoch': 0.72}\n",
      "{'loss': 1.2129, 'grad_norm': 31.46472905976925, 'learning_rate': 7.111111111111112e-06, 'epoch': 0.72}\n",
      "{'loss': 0.0, 'grad_norm': 0.0007558400303908387, 'learning_rate': 7.1075837742504415e-06, 'epoch': 0.72}\n",
      "{'loss': 0.9473, 'grad_norm': 32.947486478248614, 'learning_rate': 7.104056437389771e-06, 'epoch': 0.72}\n",
      "{'loss': 0.0031, 'grad_norm': 0.32744210570522636, 'learning_rate': 7.100529100529101e-06, 'epoch': 0.72}\n",
      "{'loss': 0.0058, 'grad_norm': 0.5762258213456911, 'learning_rate': 7.097001763668431e-06, 'epoch': 0.72}\n",
      "{'loss': 0.0134, 'grad_norm': 1.4035948376660388, 'learning_rate': 7.09347442680776e-06, 'epoch': 0.72}\n",
      "{'loss': 0.2155, 'grad_norm': 11.43308860880203, 'learning_rate': 7.08994708994709e-06, 'epoch': 0.73}\n",
      "{'loss': 0.0003, 'grad_norm': 0.03692981608360151, 'learning_rate': 7.08641975308642e-06, 'epoch': 0.73}\n",
      "{'loss': 0.0399, 'grad_norm': 4.691309893759643, 'learning_rate': 7.08289241622575e-06, 'epoch': 0.73}\n",
      "{'loss': 0.0051, 'grad_norm': 0.6600017479671104, 'learning_rate': 7.07936507936508e-06, 'epoch': 0.73}\n",
      "{'loss': 0.0397, 'grad_norm': 5.415865423167414, 'learning_rate': 7.07583774250441e-06, 'epoch': 0.73}\n",
      "{'loss': 0.0025, 'grad_norm': 0.23649778660127604, 'learning_rate': 7.0723104056437395e-06, 'epoch': 0.73}\n",
      "{'loss': 0.0, 'grad_norm': 4.2397769633938124e-05, 'learning_rate': 7.068783068783069e-06, 'epoch': 0.73}\n",
      "{'loss': 0.0, 'grad_norm': 0.0002816934430127056, 'learning_rate': 7.065255731922399e-06, 'epoch': 0.73}\n",
      "{'loss': 0.0, 'grad_norm': 0.0005174055064146763, 'learning_rate': 7.061728395061729e-06, 'epoch': 0.73}\n",
      "{'loss': 0.1896, 'grad_norm': 16.320005831249915, 'learning_rate': 7.058201058201058e-06, 'epoch': 0.73}\n",
      "{'loss': 0.0001, 'grad_norm': 0.014599018878381819, 'learning_rate': 7.054673721340388e-06, 'epoch': 0.73}\n",
      "{'loss': 0.0, 'grad_norm': 0.00028585799784259375, 'learning_rate': 7.051146384479718e-06, 'epoch': 0.73}\n",
      "{'loss': 0.016, 'grad_norm': 1.6651667123676888, 'learning_rate': 7.047619047619048e-06, 'epoch': 0.73}\n",
      "{'loss': 0.0839, 'grad_norm': 8.956749733228644, 'learning_rate': 7.044091710758378e-06, 'epoch': 0.73}\n",
      "{'loss': 0.0197, 'grad_norm': 2.440982580426738, 'learning_rate': 7.040564373897708e-06, 'epoch': 0.73}\n",
      "{'loss': 0.0001, 'grad_norm': 0.0057971153095599795, 'learning_rate': 7.0370370370370375e-06, 'epoch': 0.73}\n",
      "{'loss': 0.0, 'grad_norm': 0.0002602340019573528, 'learning_rate': 7.033509700176367e-06, 'epoch': 0.74}\n",
      "{'loss': 0.1759, 'grad_norm': 16.14519330468679, 'learning_rate': 7.029982363315697e-06, 'epoch': 0.74}\n",
      "{'loss': 0.0, 'grad_norm': 2.605259837080886e-06, 'learning_rate': 7.026455026455027e-06, 'epoch': 0.74}\n",
      "{'loss': 0.0002, 'grad_norm': 0.025852572340103224, 'learning_rate': 7.022927689594356e-06, 'epoch': 0.74}\n",
      "{'loss': 0.0003, 'grad_norm': 0.02538951999754041, 'learning_rate': 7.019400352733686e-06, 'epoch': 0.74}\n",
      "{'loss': 0.0, 'grad_norm': 1.1487750363902109e-05, 'learning_rate': 7.015873015873016e-06, 'epoch': 0.74}\n",
      "{'loss': 0.0278, 'grad_norm': 3.375252749551069, 'learning_rate': 7.012345679012347e-06, 'epoch': 0.74}\n",
      "{'loss': 0.0001, 'grad_norm': 0.008901956320846217, 'learning_rate': 7.008818342151676e-06, 'epoch': 0.74}\n",
      "{'loss': 0.0753, 'grad_norm': 6.425724852526063, 'learning_rate': 7.005291005291006e-06, 'epoch': 0.74}\n",
      "{'loss': 0.0078, 'grad_norm': 0.7506418660450337, 'learning_rate': 7.0017636684303355e-06, 'epoch': 0.74}\n",
      "{'loss': 0.0052, 'grad_norm': 0.647978309834948, 'learning_rate': 6.998236331569665e-06, 'epoch': 0.74}\n",
      "{'loss': 0.2922, 'grad_norm': 22.347534371390253, 'learning_rate': 6.994708994708995e-06, 'epoch': 0.74}\n",
      "{'loss': 0.0001, 'grad_norm': 0.01398207228631595, 'learning_rate': 6.991181657848325e-06, 'epoch': 0.74}\n",
      "{'loss': 0.0, 'grad_norm': 0.0004311354387104144, 'learning_rate': 6.987654320987654e-06, 'epoch': 0.74}\n",
      "{'loss': 0.0, 'grad_norm': 0.003987881264722657, 'learning_rate': 6.984126984126984e-06, 'epoch': 0.74}\n",
      "{'loss': 0.0008, 'grad_norm': 0.07976100514079297, 'learning_rate': 6.980599647266314e-06, 'epoch': 0.74}\n",
      "{'loss': 0.0001, 'grad_norm': 0.01311341676984332, 'learning_rate': 6.977072310405645e-06, 'epoch': 0.75}\n",
      "{'loss': 0.0, 'grad_norm': 0.00037914276066074935, 'learning_rate': 6.973544973544975e-06, 'epoch': 0.75}\n",
      "{'loss': 0.0, 'grad_norm': 0.0020270301044368485, 'learning_rate': 6.9700176366843046e-06, 'epoch': 0.75}\n",
      "{'loss': 0.0001, 'grad_norm': 0.011975701804869338, 'learning_rate': 6.966490299823634e-06, 'epoch': 0.75}\n",
      "{'loss': 0.077, 'grad_norm': 6.4133115972413615, 'learning_rate': 6.962962962962964e-06, 'epoch': 0.75}\n",
      "{'loss': 0.0002, 'grad_norm': 0.01380663842661796, 'learning_rate': 6.959435626102293e-06, 'epoch': 0.75}\n",
      "{'loss': 0.0046, 'grad_norm': 0.4824461679867232, 'learning_rate': 6.9559082892416226e-06, 'epoch': 0.75}\n",
      "{'loss': 0.0273, 'grad_norm': 2.955214795408197, 'learning_rate': 6.952380952380952e-06, 'epoch': 0.75}\n",
      "{'loss': 0.0002, 'grad_norm': 0.02506719746662877, 'learning_rate': 6.948853615520282e-06, 'epoch': 0.75}\n",
      "{'loss': 0.0, 'grad_norm': 0.0007588324795858089, 'learning_rate': 6.945326278659612e-06, 'epoch': 0.75}\n",
      "{'loss': 0.0, 'grad_norm': 0.0014629490682707563, 'learning_rate': 6.941798941798943e-06, 'epoch': 0.75}\n",
      "{'loss': 0.0001, 'grad_norm': 0.012986769175889113, 'learning_rate': 6.938271604938273e-06, 'epoch': 0.75}\n",
      "{'loss': 0.0, 'grad_norm': 0.0014282173994417935, 'learning_rate': 6.9347442680776025e-06, 'epoch': 0.75}\n",
      "{'loss': 0.0005, 'grad_norm': 0.054009065329887765, 'learning_rate': 6.931216931216932e-06, 'epoch': 0.75}\n",
      "{'loss': 0.0001, 'grad_norm': 0.013469697125266569, 'learning_rate': 6.927689594356262e-06, 'epoch': 0.75}\n",
      "{'loss': 0.0, 'grad_norm': 0.0036481849621347, 'learning_rate': 6.924162257495592e-06, 'epoch': 0.75}\n",
      "{'loss': 0.2388, 'grad_norm': 16.65793828656429, 'learning_rate': 6.920634920634921e-06, 'epoch': 0.76}\n",
      "{'loss': 0.036, 'grad_norm': 3.9414575458133783, 'learning_rate': 6.917107583774251e-06, 'epoch': 0.76}\n",
      "{'loss': 0.0, 'grad_norm': 0.0038677663111051585, 'learning_rate': 6.913580246913581e-06, 'epoch': 0.76}\n",
      "{'loss': 0.0, 'grad_norm': 0.0004980462254933868, 'learning_rate': 6.9100529100529105e-06, 'epoch': 0.76}\n",
      "{'loss': 0.0, 'grad_norm': 0.0027583781224838014, 'learning_rate': 6.906525573192241e-06, 'epoch': 0.76}\n",
      "{'loss': 0.0071, 'grad_norm': 0.9381523088194712, 'learning_rate': 6.902998236331571e-06, 'epoch': 0.76}\n",
      "{'loss': 0.033, 'grad_norm': 3.3877245895427044, 'learning_rate': 6.8994708994709005e-06, 'epoch': 0.76}\n",
      "{'loss': 0.0815, 'grad_norm': 11.15421017930544, 'learning_rate': 6.89594356261023e-06, 'epoch': 0.76}\n",
      "{'loss': 0.0001, 'grad_norm': 0.007829937250645149, 'learning_rate': 6.89241622574956e-06, 'epoch': 0.76}\n",
      "{'loss': 0.0233, 'grad_norm': 1.6352550726425372, 'learning_rate': 6.88888888888889e-06, 'epoch': 0.76}\n",
      "{'loss': 0.0, 'grad_norm': 0.00012110484941271437, 'learning_rate': 6.885361552028219e-06, 'epoch': 0.76}\n",
      "{'loss': 0.0, 'grad_norm': 1.0571337892198487e-06, 'learning_rate': 6.881834215167549e-06, 'epoch': 0.76}\n",
      "{'loss': 0.0009, 'grad_norm': 0.11937843103121688, 'learning_rate': 6.878306878306879e-06, 'epoch': 0.76}\n",
      "{'loss': 0.2343, 'grad_norm': 13.114482625746836, 'learning_rate': 6.8747795414462085e-06, 'epoch': 0.76}\n",
      "{'loss': 0.0001, 'grad_norm': 0.0035925033018242357, 'learning_rate': 6.871252204585539e-06, 'epoch': 0.76}\n",
      "{'loss': 0.0085, 'grad_norm': 1.0049684004299497, 'learning_rate': 6.867724867724869e-06, 'epoch': 0.77}\n",
      "{'loss': 0.0, 'grad_norm': 9.755338545220816e-05, 'learning_rate': 6.8641975308641985e-06, 'epoch': 0.77}\n",
      "{'loss': 0.0001, 'grad_norm': 0.009967386782267362, 'learning_rate': 6.860670194003528e-06, 'epoch': 0.77}\n",
      "{'loss': 0.0, 'grad_norm': 5.698366715222476e-06, 'learning_rate': 6.857142857142858e-06, 'epoch': 0.77}\n",
      "{'loss': 0.0, 'grad_norm': 0.00024667003272248265, 'learning_rate': 6.853615520282188e-06, 'epoch': 0.77}\n",
      "{'loss': 0.0, 'grad_norm': 0.0009430000347265999, 'learning_rate': 6.850088183421517e-06, 'epoch': 0.77}\n",
      "{'loss': 0.0, 'grad_norm': 0.0037579841077673625, 'learning_rate': 6.846560846560847e-06, 'epoch': 0.77}\n",
      "{'loss': 0.0, 'grad_norm': 4.238524700475434e-05, 'learning_rate': 6.843033509700177e-06, 'epoch': 0.77}\n",
      "{'loss': 0.0, 'grad_norm': 0.004789649979961403, 'learning_rate': 6.8395061728395065e-06, 'epoch': 0.77}\n",
      "{'loss': 0.0002, 'grad_norm': 0.022673048479843732, 'learning_rate': 6.835978835978837e-06, 'epoch': 0.77}\n",
      "{'loss': 0.0025, 'grad_norm': 0.3150416369962651, 'learning_rate': 6.832451499118167e-06, 'epoch': 0.77}\n",
      "{'loss': 0.0005, 'grad_norm': 0.04982283439268212, 'learning_rate': 6.8289241622574965e-06, 'epoch': 0.77}\n",
      "{'loss': 0.0069, 'grad_norm': 0.8015975513193379, 'learning_rate': 6.825396825396826e-06, 'epoch': 0.77}\n",
      "{'loss': 0.0001, 'grad_norm': 0.007787440526984867, 'learning_rate': 6.821869488536156e-06, 'epoch': 0.77}\n",
      "{'loss': 0.0001, 'grad_norm': 0.006028302011364759, 'learning_rate': 6.818342151675486e-06, 'epoch': 0.77}\n",
      "{'loss': 0.0001, 'grad_norm': 0.013749943871613839, 'learning_rate': 6.814814814814815e-06, 'epoch': 0.77}\n",
      "{'loss': 0.0, 'grad_norm': 0.0008979756229905207, 'learning_rate': 6.811287477954145e-06, 'epoch': 0.78}\n",
      "{'loss': 0.045, 'grad_norm': 5.154142931814393, 'learning_rate': 6.807760141093475e-06, 'epoch': 0.78}\n",
      "{'loss': 0.0, 'grad_norm': 6.256833099014245e-05, 'learning_rate': 6.8042328042328045e-06, 'epoch': 0.78}\n",
      "{'loss': 0.0023, 'grad_norm': 0.2314613272361424, 'learning_rate': 6.800705467372135e-06, 'epoch': 0.78}\n",
      "{'loss': 0.0, 'grad_norm': 0.000542507449630724, 'learning_rate': 6.797178130511465e-06, 'epoch': 0.78}\n",
      "{'loss': 0.0, 'grad_norm': 0.004802250859695032, 'learning_rate': 6.7936507936507944e-06, 'epoch': 0.78}\n",
      "{'loss': 0.0, 'grad_norm': 5.927894685800072e-05, 'learning_rate': 6.790123456790124e-06, 'epoch': 0.78}\n",
      "{'loss': 0.0, 'grad_norm': 0.00014478389049537113, 'learning_rate': 6.786596119929454e-06, 'epoch': 0.78}\n",
      "{'loss': 0.0002, 'grad_norm': 0.01953868947798883, 'learning_rate': 6.783068783068784e-06, 'epoch': 0.78}\n",
      "{'loss': 0.0, 'grad_norm': 1.3518681218792391e-05, 'learning_rate': 6.779541446208113e-06, 'epoch': 0.78}\n",
      "{'loss': 0.0, 'grad_norm': 1.2345827875742883e-05, 'learning_rate': 6.776014109347443e-06, 'epoch': 0.78}\n",
      "{'loss': 0.0097, 'grad_norm': 1.1724568766165047, 'learning_rate': 6.772486772486773e-06, 'epoch': 0.78}\n",
      "{'loss': 0.0, 'grad_norm': 0.0003962410628162639, 'learning_rate': 6.7689594356261024e-06, 'epoch': 0.78}\n",
      "{'loss': 0.0, 'grad_norm': 0.0008161729629221884, 'learning_rate': 6.765432098765433e-06, 'epoch': 0.78}\n",
      "{'loss': 0.1813, 'grad_norm': 8.450638551680715, 'learning_rate': 6.761904761904763e-06, 'epoch': 0.78}\n",
      "{'loss': 0.0, 'grad_norm': 0.0032148108004668025, 'learning_rate': 6.758377425044092e-06, 'epoch': 0.78}\n",
      "{'loss': 0.0, 'grad_norm': 0.001660583743240392, 'learning_rate': 6.754850088183422e-06, 'epoch': 0.79}\n",
      "{'loss': 0.0, 'grad_norm': 0.0004101537101701471, 'learning_rate': 6.751322751322752e-06, 'epoch': 0.79}\n",
      "{'loss': 0.0004, 'grad_norm': 0.038691363938802875, 'learning_rate': 6.7477954144620816e-06, 'epoch': 0.79}\n",
      "{'loss': 0.0, 'grad_norm': 0.0008552783277862931, 'learning_rate': 6.744268077601411e-06, 'epoch': 0.79}\n",
      "{'loss': 0.0, 'grad_norm': 1.5052511995655785e-05, 'learning_rate': 6.740740740740741e-06, 'epoch': 0.79}\n",
      "{'loss': 0.0001, 'grad_norm': 0.0050196991670432844, 'learning_rate': 6.737213403880071e-06, 'epoch': 0.79}\n",
      "{'loss': 0.1331, 'grad_norm': 9.768331656159436, 'learning_rate': 6.7336860670194e-06, 'epoch': 0.79}\n",
      "{'loss': 0.022, 'grad_norm': 2.6281115845690506, 'learning_rate': 6.730158730158731e-06, 'epoch': 0.79}\n",
      "{'loss': 0.0, 'grad_norm': 0.0009784549764056915, 'learning_rate': 6.726631393298061e-06, 'epoch': 0.79}\n",
      "{'loss': 0.0391, 'grad_norm': 4.223991472307903, 'learning_rate': 6.72310405643739e-06, 'epoch': 0.79}\n",
      "{'loss': 0.0, 'grad_norm': 0.0007381140794685477, 'learning_rate': 6.71957671957672e-06, 'epoch': 0.79}\n",
      "{'loss': 0.0, 'grad_norm': 0.003262478438724964, 'learning_rate': 6.71604938271605e-06, 'epoch': 0.79}\n",
      "{'loss': 0.0, 'grad_norm': 0.0021709490400470825, 'learning_rate': 6.7125220458553795e-06, 'epoch': 0.79}\n",
      "{'loss': 0.0, 'grad_norm': 2.8493849100643268e-05, 'learning_rate': 6.708994708994709e-06, 'epoch': 0.79}\n",
      "{'loss': 0.2681, 'grad_norm': 2.8493849100643268e-05, 'learning_rate': 6.708994708994709e-06, 'epoch': 0.79}\n",
      "{'loss': 0.1332, 'grad_norm': 14.608575972467658, 'learning_rate': 6.705467372134039e-06, 'epoch': 0.79}\n",
      "{'loss': 0.0001, 'grad_norm': 0.014198976569425733, 'learning_rate': 6.701940035273369e-06, 'epoch': 0.8}\n",
      "{'loss': 0.0036, 'grad_norm': 0.3958158259117274, 'learning_rate': 6.698412698412698e-06, 'epoch': 0.8}\n",
      "{'loss': 0.0001, 'grad_norm': 0.015126811523094889, 'learning_rate': 6.694885361552029e-06, 'epoch': 0.8}\n",
      "{'loss': 1.1719, 'grad_norm': 39.37976781649747, 'learning_rate': 6.691358024691359e-06, 'epoch': 0.8}\n",
      "{'loss': 0.0004, 'grad_norm': 0.03948877766761594, 'learning_rate': 6.687830687830688e-06, 'epoch': 0.8}\n",
      "{'loss': 0.0, 'grad_norm': 0.001612820327212316, 'learning_rate': 6.684303350970018e-06, 'epoch': 0.8}\n",
      "{'loss': 0.0, 'grad_norm': 0.0036383800064235584, 'learning_rate': 6.680776014109348e-06, 'epoch': 0.8}\n",
      "{'loss': 0.0, 'grad_norm': 3.511275330138322e-05, 'learning_rate': 6.6772486772486775e-06, 'epoch': 0.8}\n",
      "{'loss': 0.0003, 'grad_norm': 0.0284992292540727, 'learning_rate': 6.673721340388007e-06, 'epoch': 0.8}\n",
      "{'loss': 0.0048, 'grad_norm': 0.6152744825096476, 'learning_rate': 6.670194003527337e-06, 'epoch': 0.8}\n",
      "{'loss': 0.0, 'grad_norm': 0.0016529687155787188, 'learning_rate': 6.666666666666667e-06, 'epoch': 0.8}\n",
      "{'loss': 0.0, 'grad_norm': 0.0010490322112603617, 'learning_rate': 6.663139329805996e-06, 'epoch': 0.8}\n",
      "{'loss': 0.0002, 'grad_norm': 0.01503477985770592, 'learning_rate': 6.659611992945327e-06, 'epoch': 0.8}\n",
      "{'loss': 0.0, 'grad_norm': 0.001635138442828702, 'learning_rate': 6.656084656084657e-06, 'epoch': 0.8}\n",
      "{'loss': 0.0026, 'grad_norm': 0.3435264857754823, 'learning_rate': 6.652557319223986e-06, 'epoch': 0.8}\n",
      "{'loss': 0.0, 'grad_norm': 4.3983490211136904e-05, 'learning_rate': 6.649029982363316e-06, 'epoch': 0.81}\n",
      "{'loss': 0.0045, 'grad_norm': 0.7053832347658923, 'learning_rate': 6.645502645502646e-06, 'epoch': 0.81}\n",
      "{'loss': 0.0008, 'grad_norm': 0.10511859412299399, 'learning_rate': 6.6419753086419755e-06, 'epoch': 0.81}\n",
      "{'loss': 0.1285, 'grad_norm': 12.666994860228787, 'learning_rate': 6.638447971781305e-06, 'epoch': 0.81}\n",
      "{'loss': 0.0271, 'grad_norm': 2.4983075134931485, 'learning_rate': 6.634920634920635e-06, 'epoch': 0.81}\n",
      "{'loss': 0.1178, 'grad_norm': 12.356307892188608, 'learning_rate': 6.631393298059965e-06, 'epoch': 0.81}\n",
      "{'loss': 0.0006, 'grad_norm': 0.06991773958071115, 'learning_rate': 6.627865961199294e-06, 'epoch': 0.81}\n",
      "{'loss': 0.0, 'grad_norm': 0.0008418784871821943, 'learning_rate': 6.624338624338626e-06, 'epoch': 0.81}\n",
      "{'loss': 0.0007, 'grad_norm': 0.08673329521318661, 'learning_rate': 6.6208112874779555e-06, 'epoch': 0.81}\n",
      "{'loss': 0.0001, 'grad_norm': 0.010260679783635426, 'learning_rate': 6.617283950617285e-06, 'epoch': 0.81}\n",
      "{'loss': 0.0005, 'grad_norm': 0.05156289817555172, 'learning_rate': 6.613756613756615e-06, 'epoch': 0.81}\n",
      "{'loss': 0.0007, 'grad_norm': 0.06412971883601604, 'learning_rate': 6.610229276895945e-06, 'epoch': 0.81}\n",
      "{'loss': 0.0, 'grad_norm': 0.00015574390863371928, 'learning_rate': 6.6067019400352735e-06, 'epoch': 0.81}\n",
      "{'loss': 0.0009, 'grad_norm': 0.08604994375077703, 'learning_rate': 6.603174603174603e-06, 'epoch': 0.81}\n",
      "{'loss': 0.0, 'grad_norm': 0.00036063540962158805, 'learning_rate': 6.599647266313933e-06, 'epoch': 0.81}\n",
      "{'loss': 0.0013, 'grad_norm': 0.1616000654479965, 'learning_rate': 6.596119929453263e-06, 'epoch': 0.81}\n",
      "{'loss': 0.0184, 'grad_norm': 1.9475756736029002, 'learning_rate': 6.592592592592592e-06, 'epoch': 0.82}\n",
      "{'loss': 0.0, 'grad_norm': 0.0011929853436580275, 'learning_rate': 6.589065255731924e-06, 'epoch': 0.82}\n",
      "{'loss': 0.0013, 'grad_norm': 0.18767767673487618, 'learning_rate': 6.5855379188712534e-06, 'epoch': 0.82}\n",
      "{'loss': 1.1201, 'grad_norm': 31.85004192930423, 'learning_rate': 6.582010582010583e-06, 'epoch': 0.82}\n",
      "{'loss': 0.0, 'grad_norm': 0.00124257816544206, 'learning_rate': 6.578483245149913e-06, 'epoch': 0.82}\n",
      "{'loss': 0.0, 'grad_norm': 0.0007518838257224749, 'learning_rate': 6.5749559082892426e-06, 'epoch': 0.82}\n",
      "{'loss': 0.007, 'grad_norm': 0.9427058007362588, 'learning_rate': 6.571428571428572e-06, 'epoch': 0.82}\n",
      "{'loss': 0.0, 'grad_norm': 0.000817748060954049, 'learning_rate': 6.567901234567902e-06, 'epoch': 0.82}\n",
      "{'loss': 0.001, 'grad_norm': 0.11436055527995452, 'learning_rate': 6.564373897707232e-06, 'epoch': 0.82}\n",
      "{'loss': 0.0, 'grad_norm': 0.0008034050260993782, 'learning_rate': 6.560846560846561e-06, 'epoch': 0.82}\n",
      "{'loss': 0.0, 'grad_norm': 0.0021836755447270963, 'learning_rate': 6.557319223985891e-06, 'epoch': 0.82}\n",
      "{'loss': 0.0, 'grad_norm': 0.00197319617943686, 'learning_rate': 6.553791887125222e-06, 'epoch': 0.82}\n",
      "{'loss': 0.0, 'grad_norm': 0.00026150457687293784, 'learning_rate': 6.550264550264551e-06, 'epoch': 0.82}\n",
      "{'loss': 0.0, 'grad_norm': 0.002407030427874358, 'learning_rate': 6.546737213403881e-06, 'epoch': 0.82}\n",
      "{'loss': 0.0, 'grad_norm': 0.0005303564353229092, 'learning_rate': 6.543209876543211e-06, 'epoch': 0.82}\n",
      "{'loss': 0.0051, 'grad_norm': 0.5985251698631026, 'learning_rate': 6.5396825396825405e-06, 'epoch': 0.82}\n",
      "{'loss': 0.0005, 'grad_norm': 0.05052189206254432, 'learning_rate': 6.53615520282187e-06, 'epoch': 0.83}\n",
      "{'loss': 0.0, 'grad_norm': 0.00017454824091838928, 'learning_rate': 6.5326278659612e-06, 'epoch': 0.83}\n",
      "{'loss': 0.0001, 'grad_norm': 0.007407145624686701, 'learning_rate': 6.52910052910053e-06, 'epoch': 0.83}\n",
      "{'loss': 0.0001, 'grad_norm': 0.01622095312095832, 'learning_rate': 6.525573192239859e-06, 'epoch': 0.83}\n",
      "{'loss': 0.0036, 'grad_norm': 0.437417857088119, 'learning_rate': 6.522045855379189e-06, 'epoch': 0.83}\n",
      "{'loss': 0.0022, 'grad_norm': 0.35538173490831915, 'learning_rate': 6.51851851851852e-06, 'epoch': 0.83}\n",
      "{'loss': 0.0, 'grad_norm': 0.00019511713159257665, 'learning_rate': 6.514991181657849e-06, 'epoch': 0.83}\n",
      "{'loss': 0.0001, 'grad_norm': 0.007519585010101066, 'learning_rate': 6.511463844797179e-06, 'epoch': 0.83}\n",
      "{'loss': 0.0104, 'grad_norm': 1.345868458993173, 'learning_rate': 6.507936507936509e-06, 'epoch': 0.83}\n",
      "{'loss': 0.0, 'grad_norm': 0.00017988820040698035, 'learning_rate': 6.5044091710758385e-06, 'epoch': 0.83}\n",
      "{'loss': 0.0, 'grad_norm': 0.00031258070328674256, 'learning_rate': 6.500881834215168e-06, 'epoch': 0.83}\n",
      "{'loss': 0.0742, 'grad_norm': 3.803160437338108, 'learning_rate': 6.497354497354498e-06, 'epoch': 0.83}\n",
      "{'loss': 0.0025, 'grad_norm': 0.2666790189465057, 'learning_rate': 6.493827160493828e-06, 'epoch': 0.83}\n",
      "{'loss': 0.0, 'grad_norm': 0.0009560837112109342, 'learning_rate': 6.490299823633157e-06, 'epoch': 0.83}\n",
      "{'loss': 0.0, 'grad_norm': 0.0031374480555212917, 'learning_rate': 6.486772486772487e-06, 'epoch': 0.83}\n",
      "{'loss': 0.0, 'grad_norm': 1.7507020237414135e-06, 'learning_rate': 6.483245149911818e-06, 'epoch': 0.83}\n",
      "{'loss': 0.0003, 'grad_norm': 0.02805774643886728, 'learning_rate': 6.479717813051147e-06, 'epoch': 0.84}\n",
      "{'loss': 0.1648, 'grad_norm': 17.41374354799877, 'learning_rate': 6.476190476190477e-06, 'epoch': 0.84}\n",
      "{'loss': 0.0, 'grad_norm': 0.004375707399718375, 'learning_rate': 6.472663139329807e-06, 'epoch': 0.84}\n",
      "{'loss': 0.9912, 'grad_norm': 32.33952644525156, 'learning_rate': 6.4691358024691365e-06, 'epoch': 0.84}\n",
      "{'loss': 0.0, 'grad_norm': 0.00048297324047728233, 'learning_rate': 6.465608465608466e-06, 'epoch': 0.84}\n",
      "{'loss': 0.0003, 'grad_norm': 0.03785753230602456, 'learning_rate': 6.462081128747796e-06, 'epoch': 0.84}\n",
      "{'loss': 0.0, 'grad_norm': 3.7775695655885114e-05, 'learning_rate': 6.458553791887126e-06, 'epoch': 0.84}\n",
      "{'loss': 0.0002, 'grad_norm': 0.021831143509571705, 'learning_rate': 6.455026455026455e-06, 'epoch': 0.84}\n",
      "{'loss': 0.0026, 'grad_norm': 0.31971806518970536, 'learning_rate': 6.451499118165785e-06, 'epoch': 0.84}\n",
      "{'loss': 0.0, 'grad_norm': 0.0004991394737313725, 'learning_rate': 6.447971781305116e-06, 'epoch': 0.84}\n",
      "{'loss': 0.0001, 'grad_norm': 0.010883113352147327, 'learning_rate': 6.444444444444445e-06, 'epoch': 0.84}\n",
      "{'loss': 0.0001, 'grad_norm': 0.003894384741545791, 'learning_rate': 6.440917107583775e-06, 'epoch': 0.84}\n",
      "{'loss': 0.0, 'grad_norm': 9.731538933100775e-05, 'learning_rate': 6.437389770723105e-06, 'epoch': 0.84}\n",
      "{'loss': 0.0, 'grad_norm': 0.0015211124778902563, 'learning_rate': 6.4338624338624345e-06, 'epoch': 0.84}\n",
      "{'loss': 0.0, 'grad_norm': 0.00015348289197805636, 'learning_rate': 6.430335097001764e-06, 'epoch': 0.84}\n",
      "{'loss': 0.0006, 'grad_norm': 0.06343983174370159, 'learning_rate': 6.426807760141094e-06, 'epoch': 0.85}\n",
      "{'loss': 0.0001, 'grad_norm': 0.00990513158995833, 'learning_rate': 6.423280423280424e-06, 'epoch': 0.85}\n",
      "{'loss': 0.0, 'grad_norm': 0.0003112266101459883, 'learning_rate': 6.419753086419753e-06, 'epoch': 0.85}\n",
      "{'loss': 1.0391, 'grad_norm': 41.67348463098644, 'learning_rate': 6.416225749559083e-06, 'epoch': 0.85}\n",
      "{'loss': 0.0001, 'grad_norm': 0.012416951520330427, 'learning_rate': 6.412698412698414e-06, 'epoch': 0.85}\n",
      "{'loss': 0.0001, 'grad_norm': 0.009845778351733602, 'learning_rate': 6.409171075837743e-06, 'epoch': 0.85}\n",
      "{'loss': 0.0, 'grad_norm': 0.000687813510505975, 'learning_rate': 6.405643738977073e-06, 'epoch': 0.85}\n",
      "{'loss': 0.0847, 'grad_norm': 9.822319026065315, 'learning_rate': 6.402116402116403e-06, 'epoch': 0.85}\n",
      "{'loss': 0.4224, 'grad_norm': 26.52436025640805, 'learning_rate': 6.3985890652557324e-06, 'epoch': 0.85}\n",
      "{'loss': 0.0, 'grad_norm': 0.00039081261571562864, 'learning_rate': 6.395061728395062e-06, 'epoch': 0.85}\n",
      "{'loss': 0.0, 'grad_norm': 1.9214520120376235e-05, 'learning_rate': 6.391534391534392e-06, 'epoch': 0.85}\n",
      "{'loss': 0.0002, 'grad_norm': 0.025621536792446713, 'learning_rate': 6.388007054673722e-06, 'epoch': 0.85}\n",
      "{'loss': 0.0025, 'grad_norm': 0.22423537255852685, 'learning_rate': 6.384479717813051e-06, 'epoch': 0.85}\n",
      "{'loss': 0.0001, 'grad_norm': 0.01029255903229779, 'learning_rate': 6.380952380952381e-06, 'epoch': 0.85}\n",
      "{'loss': 0.0, 'grad_norm': 0.0008267011843887887, 'learning_rate': 6.3774250440917116e-06, 'epoch': 0.85}\n",
      "{'loss': 0.0002, 'grad_norm': 0.020408730907727324, 'learning_rate': 6.373897707231041e-06, 'epoch': 0.85}\n",
      "{'loss': 0.0001, 'grad_norm': 0.014217507208493147, 'learning_rate': 6.370370370370371e-06, 'epoch': 0.86}\n",
      "{'loss': 0.0, 'grad_norm': 2.458356439123871e-06, 'learning_rate': 6.366843033509701e-06, 'epoch': 0.86}\n",
      "{'loss': 0.0612, 'grad_norm': 7.660576792781057, 'learning_rate': 6.36331569664903e-06, 'epoch': 0.86}\n",
      "{'loss': 0.0001, 'grad_norm': 0.013434584994168393, 'learning_rate': 6.35978835978836e-06, 'epoch': 0.86}\n",
      "{'loss': 0.0021, 'grad_norm': 0.2940361794986855, 'learning_rate': 6.35626102292769e-06, 'epoch': 0.86}\n",
      "{'loss': 0.0002, 'grad_norm': 0.028007224675651074, 'learning_rate': 6.3527336860670196e-06, 'epoch': 0.86}\n",
      "{'loss': 0.0, 'grad_norm': 0.0019215154243078043, 'learning_rate': 6.349206349206349e-06, 'epoch': 0.86}\n",
      "{'loss': 0.0006, 'grad_norm': 0.07215208400130005, 'learning_rate': 6.345679012345679e-06, 'epoch': 0.86}\n",
      "{'loss': 0.0093, 'grad_norm': 0.9380338738167091, 'learning_rate': 6.3421516754850095e-06, 'epoch': 0.86}\n",
      "{'loss': 0.0, 'grad_norm': 6.45935139423181e-05, 'learning_rate': 6.338624338624339e-06, 'epoch': 0.86}\n",
      "{'loss': 0.0001, 'grad_norm': 0.007205813695631288, 'learning_rate': 6.335097001763669e-06, 'epoch': 0.86}\n",
      "{'loss': 0.0034, 'grad_norm': 0.4945854777996918, 'learning_rate': 6.331569664902999e-06, 'epoch': 0.86}\n",
      "{'loss': 0.0002, 'grad_norm': 0.016192572565199483, 'learning_rate': 6.328042328042328e-06, 'epoch': 0.86}\n",
      "{'loss': 0.0013, 'grad_norm': 0.15037454791647162, 'learning_rate': 6.324514991181658e-06, 'epoch': 0.86}\n",
      "{'loss': 0.0251, 'grad_norm': 3.278809720832364, 'learning_rate': 6.320987654320988e-06, 'epoch': 0.86}\n",
      "{'loss': 0.0, 'grad_norm': 0.001729859600712946, 'learning_rate': 6.3174603174603175e-06, 'epoch': 0.86}\n",
      "{'loss': 0.0042, 'grad_norm': 0.4974184596988603, 'learning_rate': 6.313932980599647e-06, 'epoch': 0.87}\n",
      "{'loss': 0.0, 'grad_norm': 0.0005149861176770349, 'learning_rate': 6.310405643738977e-06, 'epoch': 0.87}\n",
      "{'loss': 0.0, 'grad_norm': 0.0002229448136585216, 'learning_rate': 6.3068783068783075e-06, 'epoch': 0.87}\n",
      "{'loss': 0.0025, 'grad_norm': 0.3415262077015871, 'learning_rate': 6.303350970017637e-06, 'epoch': 0.87}\n",
      "{'loss': 0.0002, 'grad_norm': 0.022220510392511227, 'learning_rate': 6.299823633156967e-06, 'epoch': 0.87}\n",
      "{'loss': 0.0029, 'grad_norm': 0.2932420981262079, 'learning_rate': 6.296296296296297e-06, 'epoch': 0.87}\n",
      "{'loss': 0.0, 'grad_norm': 5.884057806398214e-05, 'learning_rate': 6.292768959435626e-06, 'epoch': 0.87}\n",
      "{'loss': 0.0001, 'grad_norm': 0.007787109770854364, 'learning_rate': 6.289241622574956e-06, 'epoch': 0.87}\n",
      "{'loss': 0.009, 'grad_norm': 1.3201066257298681, 'learning_rate': 6.285714285714286e-06, 'epoch': 0.87}\n",
      "{'loss': 0.0, 'grad_norm': 1.717617315809456e-05, 'learning_rate': 6.2821869488536155e-06, 'epoch': 0.87}\n",
      "{'loss': 0.0, 'grad_norm': 3.713027769190574e-05, 'learning_rate': 6.278659611992945e-06, 'epoch': 0.87}\n",
      "{'loss': 0.0, 'grad_norm': 8.444610490207561e-07, 'learning_rate': 6.275132275132275e-06, 'epoch': 0.87}\n",
      "{'loss': 0.3345, 'grad_norm': 14.37816445723096, 'learning_rate': 6.271604938271606e-06, 'epoch': 0.87}\n",
      "{'loss': 0.0001, 'grad_norm': 0.007220794949248867, 'learning_rate': 6.268077601410936e-06, 'epoch': 0.87}\n",
      "{'loss': 0.0, 'grad_norm': 0.00019723545316286526, 'learning_rate': 6.264550264550266e-06, 'epoch': 0.87}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 44%|████▎     | 1377/3150 [05:55<08:04,  3.66it/s]  "
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.0159, 'grad_norm': 1.7536696378495038, 'learning_rate': 6.2610229276895955e-06, 'epoch': 0.87}\n",
      "{'loss': 0.0004, 'grad_norm': 0.03909537600833843, 'learning_rate': 6.257495590828925e-06, 'epoch': 0.88}\n",
      "{'loss': 0.0, 'grad_norm': 1.3072261291430532e-05, 'learning_rate': 6.253968253968254e-06, 'epoch': 0.88}\n",
      "{'loss': 0.0137, 'grad_norm': 0.9366626178635848, 'learning_rate': 6.250440917107584e-06, 'epoch': 0.88}\n",
      "{'loss': 0.0006, 'grad_norm': 0.06750650731592978, 'learning_rate': 6.2469135802469135e-06, 'epoch': 0.88}\n",
      "{'loss': 0.0066, 'grad_norm': 0.6520149178816838, 'learning_rate': 6.243386243386243e-06, 'epoch': 0.88}\n",
      "{'loss': 0.0001, 'grad_norm': 0.007519813360458526, 'learning_rate': 6.239858906525573e-06, 'epoch': 0.88}\n",
      "{'loss': 0.3818, 'grad_norm': 22.781509879347606, 'learning_rate': 6.236331569664904e-06, 'epoch': 0.88}\n",
      "{'loss': 0.0, 'grad_norm': 0.0004221153474469201, 'learning_rate': 6.232804232804234e-06, 'epoch': 0.88}\n",
      "{'loss': 0.0109, 'grad_norm': 1.572822121648162, 'learning_rate': 6.229276895943564e-06, 'epoch': 0.88}\n",
      "{'loss': 0.0006, 'grad_norm': 0.07694564797534252, 'learning_rate': 6.2257495590828935e-06, 'epoch': 0.88}\n",
      "{'loss': 0.0, 'grad_norm': 0.00028936206682971026, 'learning_rate': 6.222222222222223e-06, 'epoch': 0.88}\n",
      "{'loss': 0.0003, 'grad_norm': 0.033163957424356814, 'learning_rate': 6.218694885361553e-06, 'epoch': 0.88}\n",
      "{'loss': 0.0, 'grad_norm': 0.00034297027433202135, 'learning_rate': 6.215167548500883e-06, 'epoch': 0.88}\n",
      "{'loss': 0.0, 'grad_norm': 0.0035883481131207115, 'learning_rate': 6.211640211640212e-06, 'epoch': 0.88}\n",
      "{'loss': 0.0001, 'grad_norm': 0.006395607603385, 'learning_rate': 6.208112874779542e-06, 'epoch': 0.88}\n",
      "{'loss': 0.0, 'grad_norm': 0.002213270168573042, 'learning_rate': 6.204585537918871e-06, 'epoch': 0.89}\n",
      "{'loss': 0.0008, 'grad_norm': 0.11837623400583744, 'learning_rate': 6.201058201058202e-06, 'epoch': 0.89}\n",
      "{'loss': 0.0, 'grad_norm': 0.0030002093811133993, 'learning_rate': 6.197530864197532e-06, 'epoch': 0.89}\n",
      "{'loss': 0.0, 'grad_norm': 0.0009946665963772006, 'learning_rate': 6.194003527336862e-06, 'epoch': 0.89}\n",
      "{'loss': 0.0002, 'grad_norm': 0.018327588257497508, 'learning_rate': 6.1904761904761914e-06, 'epoch': 0.89}\n",
      "{'loss': 0.0002, 'grad_norm': 0.023684303635379485, 'learning_rate': 6.186948853615521e-06, 'epoch': 0.89}\n",
      "{'loss': 0.0001, 'grad_norm': 0.01682003975811877, 'learning_rate': 6.183421516754851e-06, 'epoch': 0.89}\n",
      "{'loss': 0.0002, 'grad_norm': 0.022434666290048012, 'learning_rate': 6.1798941798941806e-06, 'epoch': 0.89}\n",
      "{'loss': 0.3228, 'grad_norm': 22.04439919545715, 'learning_rate': 6.17636684303351e-06, 'epoch': 0.89}\n",
      "{'loss': 0.0, 'grad_norm': 0.00020560302331354454, 'learning_rate': 6.17283950617284e-06, 'epoch': 0.89}\n",
      "{'loss': 0.0, 'grad_norm': 8.706993689479989e-05, 'learning_rate': 6.16931216931217e-06, 'epoch': 0.89}\n",
      "{'loss': 0.3428, 'grad_norm': 21.561197169653447, 'learning_rate': 6.1657848324515e-06, 'epoch': 0.89}\n",
      "{'loss': 0.0001, 'grad_norm': 0.009356668970567117, 'learning_rate': 6.16225749559083e-06, 'epoch': 0.89}\n",
      "{'loss': 0.0106, 'grad_norm': 1.0568392244371416, 'learning_rate': 6.15873015873016e-06, 'epoch': 0.89}\n",
      "{'loss': 0.1631, 'grad_norm': 13.099564383449277, 'learning_rate': 6.155202821869489e-06, 'epoch': 0.89}\n",
      "{'loss': 0.002, 'grad_norm': 0.23991840797286848, 'learning_rate': 6.151675485008819e-06, 'epoch': 0.89}\n",
      "{'loss': 0.8276, 'grad_norm': 34.79816770443435, 'learning_rate': 6.148148148148149e-06, 'epoch': 0.9}\n",
      "{'loss': 0.0, 'grad_norm': 0.0004063081935517296, 'learning_rate': 6.1446208112874785e-06, 'epoch': 0.9}\n",
      "{'loss': 1.9775, 'grad_norm': 30.64631851265183, 'learning_rate': 6.141093474426808e-06, 'epoch': 0.9}\n",
      "{'loss': 0.0, 'grad_norm': 8.462654327765954e-06, 'learning_rate': 6.137566137566138e-06, 'epoch': 0.9}\n",
      "{'loss': 0.0, 'grad_norm': 0.0014338373622007515, 'learning_rate': 6.134038800705468e-06, 'epoch': 0.9}\n",
      "{'loss': 0.0, 'grad_norm': 4.209720823898793e-06, 'learning_rate': 6.130511463844798e-06, 'epoch': 0.9}\n",
      "{'loss': 0.0, 'grad_norm': 3.583493195211275e-05, 'learning_rate': 6.126984126984128e-06, 'epoch': 0.9}\n",
      "{'loss': 0.0, 'grad_norm': 7.669957268328695e-05, 'learning_rate': 6.123456790123458e-06, 'epoch': 0.9}\n",
      "{'loss': 0.0005, 'grad_norm': 0.07942694392999808, 'learning_rate': 6.119929453262787e-06, 'epoch': 0.9}\n",
      "{'loss': 0.0, 'grad_norm': 0.0001756344692910597, 'learning_rate': 6.116402116402117e-06, 'epoch': 0.9}\n",
      "{'loss': 0.0, 'grad_norm': 0.003111611169981156, 'learning_rate': 6.112874779541447e-06, 'epoch': 0.9}\n",
      "{'loss': 0.0009, 'grad_norm': 0.09914724352683323, 'learning_rate': 6.1093474426807765e-06, 'epoch': 0.9}\n",
      "{'loss': 0.0003, 'grad_norm': 0.030060144606582885, 'learning_rate': 6.105820105820106e-06, 'epoch': 0.9}\n",
      "{'loss': 0.0, 'grad_norm': 9.51866710647572e-06, 'learning_rate': 6.102292768959436e-06, 'epoch': 0.9}\n",
      "{'loss': 0.4316, 'grad_norm': 21.395092182143593, 'learning_rate': 6.098765432098766e-06, 'epoch': 0.9}\n",
      "{'loss': 0.0005, 'grad_norm': 0.05287426038234921, 'learning_rate': 6.095238095238096e-06, 'epoch': 0.9}\n",
      "{'loss': 0.0024, 'grad_norm': 0.2667109925092273, 'learning_rate': 6.091710758377426e-06, 'epoch': 0.91}\n",
      "{'loss': 0.0003, 'grad_norm': 0.030844658914503633, 'learning_rate': 6.088183421516756e-06, 'epoch': 0.91}\n",
      "{'loss': 0.0, 'grad_norm': 6.047638794742902e-05, 'learning_rate': 6.084656084656085e-06, 'epoch': 0.91}\n",
      "{'loss': 0.0, 'grad_norm': 0.0001262551812841917, 'learning_rate': 6.081128747795415e-06, 'epoch': 0.91}\n",
      "{'loss': 0.0654, 'grad_norm': 5.275302404621471, 'learning_rate': 6.077601410934745e-06, 'epoch': 0.91}\n",
      "{'loss': 0.0, 'grad_norm': 0.0004935918755414255, 'learning_rate': 6.0740740740740745e-06, 'epoch': 0.91}\n",
      "{'loss': 0.0, 'grad_norm': 0.002125282738989415, 'learning_rate': 6.070546737213404e-06, 'epoch': 0.91}\n",
      "{'loss': 0.0, 'grad_norm': 0.0003416060390484243, 'learning_rate': 6.067019400352734e-06, 'epoch': 0.91}\n",
      "{'loss': 0.0001, 'grad_norm': 0.011348910931247801, 'learning_rate': 6.063492063492064e-06, 'epoch': 0.91}\n",
      "{'loss': 0.0007, 'grad_norm': 0.09385535658128984, 'learning_rate': 6.059964726631394e-06, 'epoch': 0.91}\n",
      "{'loss': 0.0, 'grad_norm': 0.0036356597894207482, 'learning_rate': 6.056437389770724e-06, 'epoch': 0.91}\n",
      "{'loss': 0.0, 'grad_norm': 0.00192254060201484, 'learning_rate': 6.052910052910054e-06, 'epoch': 0.91}\n",
      "{'loss': 0.0, 'grad_norm': 5.213619888380536e-05, 'learning_rate': 6.049382716049383e-06, 'epoch': 0.91}\n",
      "{'loss': 0.0007, 'grad_norm': 0.06500855896150665, 'learning_rate': 6.045855379188713e-06, 'epoch': 0.91}\n",
      "{'loss': 0.0, 'grad_norm': 4.710760127789827e-06, 'learning_rate': 6.042328042328043e-06, 'epoch': 0.91}\n",
      "{'loss': 0.0012, 'grad_norm': 0.17209804216216892, 'learning_rate': 6.0388007054673725e-06, 'epoch': 0.91}\n",
      "{'loss': 0.0, 'grad_norm': 2.5506338769803994e-05, 'learning_rate': 6.035273368606702e-06, 'epoch': 0.92}\n",
      "{'loss': 0.0, 'grad_norm': 0.004636778060598326, 'learning_rate': 6.031746031746032e-06, 'epoch': 0.92}\n",
      "{'loss': 0.0, 'grad_norm': 0.00015717471326141686, 'learning_rate': 6.028218694885362e-06, 'epoch': 0.92}\n",
      "{'loss': 0.0, 'grad_norm': 0.00032611895810546424, 'learning_rate': 6.024691358024692e-06, 'epoch': 0.92}\n",
      "{'loss': 0.0601, 'grad_norm': 7.805917637196245, 'learning_rate': 6.021164021164022e-06, 'epoch': 0.92}\n",
      "{'loss': 0.0427, 'grad_norm': 4.756464925514166, 'learning_rate': 6.017636684303352e-06, 'epoch': 0.92}\n",
      "{'loss': 0.0026, 'grad_norm': 0.3088175408817194, 'learning_rate': 6.014109347442681e-06, 'epoch': 0.92}\n",
      "{'loss': 0.0, 'grad_norm': 0.0004364761655183345, 'learning_rate': 6.010582010582011e-06, 'epoch': 0.92}\n",
      "{'loss': 0.118, 'grad_norm': 9.154903377937694, 'learning_rate': 6.007054673721341e-06, 'epoch': 0.92}\n",
      "{'loss': 0.0183, 'grad_norm': 1.9959536986666473, 'learning_rate': 6.0035273368606704e-06, 'epoch': 0.92}\n",
      "{'loss': 0.0155, 'grad_norm': 1.784115811578939, 'learning_rate': 6e-06, 'epoch': 0.92}\n",
      "{'loss': 0.1058, 'grad_norm': 9.191119045661601, 'learning_rate': 5.99647266313933e-06, 'epoch': 0.92}\n",
      "{'loss': 0.0894, 'grad_norm': 4.3534192382219175, 'learning_rate': 5.99294532627866e-06, 'epoch': 0.92}\n",
      "{'loss': 0.0001, 'grad_norm': 0.006569737150649437, 'learning_rate': 5.989417989417989e-06, 'epoch': 0.92}\n",
      "{'loss': 1.0273, 'grad_norm': 27.66448169337863, 'learning_rate': 5.98589065255732e-06, 'epoch': 0.92}\n",
      "{'loss': 0.4526, 'grad_norm': 22.11994764325256, 'learning_rate': 5.9823633156966496e-06, 'epoch': 0.93}\n",
      "{'loss': 0.0, 'grad_norm': 0.00023083278328179507, 'learning_rate': 5.978835978835979e-06, 'epoch': 0.93}\n",
      "{'loss': 0.0, 'grad_norm': 0.000338780898640797, 'learning_rate': 5.975308641975309e-06, 'epoch': 0.93}\n",
      "{'loss': 0.0, 'grad_norm': 0.004297675084001466, 'learning_rate': 5.971781305114639e-06, 'epoch': 0.93}\n",
      "{'loss': 0.0155, 'grad_norm': 2.421232175116441, 'learning_rate': 5.968253968253968e-06, 'epoch': 0.93}\n",
      "{'loss': 0.0003, 'grad_norm': 0.04289022091768996, 'learning_rate': 5.964726631393298e-06, 'epoch': 0.93}\n",
      "{'loss': 0.0, 'grad_norm': 0.00010189797550971045, 'learning_rate': 5.961199294532628e-06, 'epoch': 0.93}\n",
      "{'loss': 0.0, 'grad_norm': 0.00029340481897211277, 'learning_rate': 5.9576719576719576e-06, 'epoch': 0.93}\n",
      "{'loss': 0.0001, 'grad_norm': 0.006298412045424221, 'learning_rate': 5.954144620811287e-06, 'epoch': 0.93}\n",
      "{'loss': 0.0001, 'grad_norm': 0.008872961649447633, 'learning_rate': 5.950617283950618e-06, 'epoch': 0.93}\n",
      "{'loss': 0.0005, 'grad_norm': 0.045942290641796814, 'learning_rate': 5.9470899470899475e-06, 'epoch': 0.93}\n",
      "{'loss': 0.0, 'grad_norm': 3.0894705470515196e-05, 'learning_rate': 5.943562610229277e-06, 'epoch': 0.93}\n",
      "{'loss': 0.7031, 'grad_norm': 19.840528620168726, 'learning_rate': 5.940035273368607e-06, 'epoch': 0.93}\n",
      "{'loss': 0.0384, 'grad_norm': 4.596404469964939, 'learning_rate': 5.936507936507937e-06, 'epoch': 0.93}\n",
      "{'loss': 0.0034, 'grad_norm': 0.2861676764486142, 'learning_rate': 5.932980599647266e-06, 'epoch': 0.93}\n",
      "{'loss': 0.0017, 'grad_norm': 0.1959446975079634, 'learning_rate': 5.929453262786596e-06, 'epoch': 0.93}\n",
      "{'loss': 0.0002, 'grad_norm': 0.018060842830192197, 'learning_rate': 5.925925925925926e-06, 'epoch': 0.94}\n",
      "{'loss': 0.0117, 'grad_norm': 1.4256008086269667, 'learning_rate': 5.9223985890652555e-06, 'epoch': 0.94}\n",
      "{'loss': 0.0001, 'grad_norm': 0.00558341902365659, 'learning_rate': 5.918871252204585e-06, 'epoch': 0.94}\n",
      "{'loss': 0.0, 'grad_norm': 1.5804547493817094e-05, 'learning_rate': 5.915343915343917e-06, 'epoch': 0.94}\n",
      "{'loss': 0.0, 'grad_norm': 0.0029857742659801837, 'learning_rate': 5.911816578483246e-06, 'epoch': 0.94}\n",
      "{'loss': 0.1814, 'grad_norm': 17.444318853251517, 'learning_rate': 5.908289241622576e-06, 'epoch': 0.94}\n",
      "{'loss': 0.0008, 'grad_norm': 0.11014313129333965, 'learning_rate': 5.904761904761905e-06, 'epoch': 0.94}\n",
      "{'loss': 0.0, 'grad_norm': 0.0014284293598795794, 'learning_rate': 5.901234567901235e-06, 'epoch': 0.94}\n",
      "{'loss': 0.0001, 'grad_norm': 0.0112208754314194, 'learning_rate': 5.897707231040564e-06, 'epoch': 0.94}\n",
      "{'loss': 0.0006, 'grad_norm': 0.07225347429657751, 'learning_rate': 5.894179894179894e-06, 'epoch': 0.94}\n",
      "{'loss': 0.5747, 'grad_norm': 34.97353037867978, 'learning_rate': 5.890652557319224e-06, 'epoch': 0.94}\n",
      "{'loss': 0.0, 'grad_norm': 0.0009191045501348066, 'learning_rate': 5.8871252204585535e-06, 'epoch': 0.94}\n",
      "{'loss': 0.0, 'grad_norm': 2.533272643798383e-06, 'learning_rate': 5.883597883597883e-06, 'epoch': 0.94}\n",
      "{'loss': 0.0, 'grad_norm': 0.0035793390531089803, 'learning_rate': 5.880070546737215e-06, 'epoch': 0.94}\n",
      "{'loss': 0.0, 'grad_norm': 0.005544934431767598, 'learning_rate': 5.876543209876544e-06, 'epoch': 0.94}\n",
      "{'loss': 0.0015, 'grad_norm': 0.16072577303465987, 'learning_rate': 5.873015873015874e-06, 'epoch': 0.94}\n",
      "{'loss': 0.0001, 'grad_norm': 0.019169015836267737, 'learning_rate': 5.869488536155204e-06, 'epoch': 0.95}\n",
      "{'loss': 0.012, 'grad_norm': 1.743810060622101, 'learning_rate': 5.8659611992945335e-06, 'epoch': 0.95}\n",
      "{'loss': 0.0025, 'grad_norm': 0.26093468007831094, 'learning_rate': 5.862433862433863e-06, 'epoch': 0.95}\n",
      "{'loss': 0.0007, 'grad_norm': 0.09329680337106143, 'learning_rate': 5.858906525573193e-06, 'epoch': 0.95}\n",
      "{'loss': 0.0, 'grad_norm': 0.0001553068705508635, 'learning_rate': 5.855379188712523e-06, 'epoch': 0.95}\n",
      "{'loss': 0.0, 'grad_norm': 5.993665672744202e-05, 'learning_rate': 5.8518518518518515e-06, 'epoch': 0.95}\n",
      "{'loss': 0.0001, 'grad_norm': 0.0067773483775175275, 'learning_rate': 5.848324514991181e-06, 'epoch': 0.95}\n",
      "{'loss': 0.0001, 'grad_norm': 0.010013289674446355, 'learning_rate': 5.844797178130513e-06, 'epoch': 0.95}\n",
      "{'loss': 0.0001, 'grad_norm': 0.011304952370999755, 'learning_rate': 5.841269841269842e-06, 'epoch': 0.95}\n",
      "{'loss': 0.0, 'grad_norm': 0.005169762299092592, 'learning_rate': 5.837742504409172e-06, 'epoch': 0.95}\n",
      "{'loss': 0.0047, 'grad_norm': 0.48558289140358546, 'learning_rate': 5.834215167548502e-06, 'epoch': 0.95}\n",
      "{'loss': 0.0007, 'grad_norm': 0.07696770975189779, 'learning_rate': 5.8306878306878314e-06, 'epoch': 0.95}\n",
      "{'loss': 0.0025, 'grad_norm': 0.3164485385091337, 'learning_rate': 5.827160493827161e-06, 'epoch': 0.95}\n",
      "{'loss': 0.0218, 'grad_norm': 2.82897396938933, 'learning_rate': 5.823633156966491e-06, 'epoch': 0.95}\n",
      "{'loss': 0.0, 'grad_norm': 0.0012081080405812047, 'learning_rate': 5.820105820105821e-06, 'epoch': 0.95}\n",
      "{'loss': 0.0032, 'grad_norm': 0.3652273779214137, 'learning_rate': 5.81657848324515e-06, 'epoch': 0.95}\n",
      "{'loss': 0.0, 'grad_norm': 8.17938657385526e-07, 'learning_rate': 5.81305114638448e-06, 'epoch': 0.96}\n",
      "{'loss': 0.0003, 'grad_norm': 0.03889323007846274, 'learning_rate': 5.8095238095238106e-06, 'epoch': 0.96}\n",
      "{'loss': 0.0113, 'grad_norm': 1.5149256085553466, 'learning_rate': 5.80599647266314e-06, 'epoch': 0.96}\n",
      "{'loss': 0.0004, 'grad_norm': 0.04259995395282654, 'learning_rate': 5.80246913580247e-06, 'epoch': 0.96}\n",
      "{'loss': 0.8091, 'grad_norm': 35.017913593772796, 'learning_rate': 5.7989417989418e-06, 'epoch': 0.96}\n",
      "{'loss': 0.0001, 'grad_norm': 0.027298689012445258, 'learning_rate': 5.795414462081129e-06, 'epoch': 0.96}\n",
      "{'loss': 0.0026, 'grad_norm': 0.3516974402114878, 'learning_rate': 5.791887125220459e-06, 'epoch': 0.96}\n",
      "{'loss': 0.0723, 'grad_norm': 4.488729987358792, 'learning_rate': 5.788359788359789e-06, 'epoch': 0.96}\n",
      "{'loss': 0.2484, 'grad_norm': 12.21107932879086, 'learning_rate': 5.7848324514991186e-06, 'epoch': 0.96}\n",
      "{'loss': 1.667, 'grad_norm': 37.55425274737952, 'learning_rate': 5.781305114638448e-06, 'epoch': 0.96}\n",
      "{'loss': 0.0011, 'grad_norm': 0.12222138021716727, 'learning_rate': 5.777777777777778e-06, 'epoch': 0.96}\n",
      "{'loss': 0.0, 'grad_norm': 2.8236542428666193e-06, 'learning_rate': 5.7742504409171085e-06, 'epoch': 0.96}\n",
      "{'loss': 0.0021, 'grad_norm': 0.23830122942101112, 'learning_rate': 5.770723104056438e-06, 'epoch': 0.96}\n",
      "{'loss': 0.0, 'grad_norm': 0.0010203437232636737, 'learning_rate': 5.767195767195768e-06, 'epoch': 0.96}\n",
      "{'loss': 0.0001, 'grad_norm': 0.007098021563371264, 'learning_rate': 5.763668430335098e-06, 'epoch': 0.96}\n",
      "{'loss': 0.0135, 'grad_norm': 1.4168571549794908, 'learning_rate': 5.760141093474427e-06, 'epoch': 0.97}\n",
      "{'loss': 0.0018, 'grad_norm': 0.2344929160088199, 'learning_rate': 5.756613756613757e-06, 'epoch': 0.97}\n",
      "{'loss': 0.0894, 'grad_norm': 5.269837457513589, 'learning_rate': 5.753086419753087e-06, 'epoch': 0.97}\n",
      "{'loss': 0.0076, 'grad_norm': 1.0145826419355373, 'learning_rate': 5.7495590828924165e-06, 'epoch': 0.97}\n",
      "{'loss': 0.0, 'grad_norm': 8.335581576433923e-06, 'learning_rate': 5.746031746031746e-06, 'epoch': 0.97}\n",
      "{'loss': 0.0, 'grad_norm': 0.0036011636246072987, 'learning_rate': 5.742504409171076e-06, 'epoch': 0.97}\n",
      "{'loss': 0.0003, 'grad_norm': 0.019534578635767415, 'learning_rate': 5.7389770723104065e-06, 'epoch': 0.97}\n",
      "{'loss': 0.0001, 'grad_norm': 0.006729551980856392, 'learning_rate': 5.735449735449736e-06, 'epoch': 0.97}\n",
      "{'loss': 0.7739, 'grad_norm': 30.26568449110054, 'learning_rate': 5.731922398589066e-06, 'epoch': 0.97}\n",
      "{'loss': 0.0003, 'grad_norm': 0.02388542284456296, 'learning_rate': 5.728395061728396e-06, 'epoch': 0.97}\n",
      "{'loss': 0.0, 'grad_norm': 0.001353159679357558, 'learning_rate': 5.724867724867725e-06, 'epoch': 0.97}\n",
      "{'loss': 0.0007, 'grad_norm': 0.09932764734288656, 'learning_rate': 5.721340388007055e-06, 'epoch': 0.97}\n",
      "{'loss': 0.0001, 'grad_norm': 0.01493944499211583, 'learning_rate': 5.717813051146385e-06, 'epoch': 0.97}\n",
      "{'loss': 0.0005, 'grad_norm': 0.03247483843807909, 'learning_rate': 5.7142857142857145e-06, 'epoch': 0.97}\n",
      "{'loss': 0.0, 'grad_norm': 0.0034713561717796032, 'learning_rate': 5.710758377425044e-06, 'epoch': 0.97}\n",
      "{'loss': 0.0, 'grad_norm': 0.0006054286449828665, 'learning_rate': 5.707231040564374e-06, 'epoch': 0.97}\n",
      "{'loss': 0.0006, 'grad_norm': 0.055448423946283396, 'learning_rate': 5.7037037037037045e-06, 'epoch': 0.98}\n",
      "{'loss': 0.0, 'grad_norm': 7.400665635486355e-06, 'learning_rate': 5.700176366843034e-06, 'epoch': 0.98}\n",
      "{'loss': 0.0042, 'grad_norm': 0.6171120464703217, 'learning_rate': 5.696649029982364e-06, 'epoch': 0.98}\n",
      "{'loss': 0.0, 'grad_norm': 0.0011076644425020226, 'learning_rate': 5.693121693121694e-06, 'epoch': 0.98}\n",
      "{'loss': 0.0001, 'grad_norm': 0.00594974713963548, 'learning_rate': 5.689594356261023e-06, 'epoch': 0.98}\n",
      "{'loss': 0.0, 'grad_norm': 0.002922587496160047, 'learning_rate': 5.686067019400353e-06, 'epoch': 0.98}\n",
      "{'loss': 0.0001, 'grad_norm': 0.011976530488063821, 'learning_rate': 5.682539682539683e-06, 'epoch': 0.98}\n",
      "{'loss': 0.0022, 'grad_norm': 0.19665839904915422, 'learning_rate': 5.6790123456790125e-06, 'epoch': 0.98}\n",
      "{'loss': 0.0008, 'grad_norm': 0.08494768368174312, 'learning_rate': 5.675485008818342e-06, 'epoch': 0.98}\n",
      "{'loss': 0.0, 'grad_norm': 7.404188078743237e-06, 'learning_rate': 5.671957671957672e-06, 'epoch': 0.98}\n",
      "{'loss': 0.002, 'grad_norm': 0.27111637779585185, 'learning_rate': 5.6684303350970025e-06, 'epoch': 0.98}\n",
      "{'loss': 0.0, 'grad_norm': 3.3085380105753234e-05, 'learning_rate': 5.664902998236332e-06, 'epoch': 0.98}\n",
      "{'loss': 0.0001, 'grad_norm': 0.010150251041064226, 'learning_rate': 5.661375661375662e-06, 'epoch': 0.98}\n",
      "{'loss': 0.0003, 'grad_norm': 0.028392350719900785, 'learning_rate': 5.657848324514992e-06, 'epoch': 0.98}\n",
      "{'loss': 0.0027, 'grad_norm': 0.28791093845208127, 'learning_rate': 5.654320987654321e-06, 'epoch': 0.98}\n",
      "{'loss': 0.0091, 'grad_norm': 1.1378130806803826, 'learning_rate': 5.650793650793651e-06, 'epoch': 0.98}\n",
      "{'loss': 0.0005, 'grad_norm': 0.05941408221616299, 'learning_rate': 5.647266313932981e-06, 'epoch': 0.99}\n",
      "{'loss': 0.0044, 'grad_norm': 0.46778702367751157, 'learning_rate': 5.6437389770723105e-06, 'epoch': 0.99}\n",
      "{'loss': 0.0018, 'grad_norm': 0.18695738877225818, 'learning_rate': 5.64021164021164e-06, 'epoch': 0.99}\n",
      "{'loss': 0.0, 'grad_norm': 2.610812305490361e-06, 'learning_rate': 5.63668430335097e-06, 'epoch': 0.99}\n",
      "{'loss': 0.0003, 'grad_norm': 0.03400592915161539, 'learning_rate': 5.6331569664903004e-06, 'epoch': 0.99}\n",
      "{'loss': 1.6797, 'grad_norm': 21.562206321596083, 'learning_rate': 5.62962962962963e-06, 'epoch': 0.99}\n",
      "{'loss': 0.0003, 'grad_norm': 0.0385646113974211, 'learning_rate': 5.62610229276896e-06, 'epoch': 0.99}\n",
      "{'loss': 0.0, 'grad_norm': 0.0032279579385206094, 'learning_rate': 5.62257495590829e-06, 'epoch': 0.99}\n",
      "{'loss': 0.0, 'grad_norm': 0.00041126831716420045, 'learning_rate': 5.619047619047619e-06, 'epoch': 0.99}\n",
      "{'loss': 0.0, 'grad_norm': 0.002220727045245227, 'learning_rate': 5.615520282186949e-06, 'epoch': 0.99}\n",
      "{'loss': 0.0, 'grad_norm': 5.242603988562092e-05, 'learning_rate': 5.611992945326279e-06, 'epoch': 0.99}\n",
      "{'loss': 0.0, 'grad_norm': 0.00047446645065792536, 'learning_rate': 5.6084656084656084e-06, 'epoch': 0.99}\n",
      "{'loss': 0.0, 'grad_norm': 0.000274686930337093, 'learning_rate': 5.604938271604938e-06, 'epoch': 0.99}\n",
      "{'loss': 0.0002, 'grad_norm': 0.0160169512574653, 'learning_rate': 5.601410934744268e-06, 'epoch': 0.99}\n",
      "{'loss': 0.0, 'grad_norm': 7.502171147008243e-07, 'learning_rate': 5.597883597883598e-06, 'epoch': 0.99}\n",
      "{'loss': 0.0003, 'grad_norm': 0.03822289350559787, 'learning_rate': 5.594356261022928e-06, 'epoch': 0.99}\n",
      "{'loss': 0.0, 'grad_norm': 0.0007156002089380734, 'learning_rate': 5.590828924162258e-06, 'epoch': 1.0}\n",
      "{'loss': 0.0007, 'grad_norm': 0.07662010057550996, 'learning_rate': 5.5873015873015876e-06, 'epoch': 1.0}\n",
      "{'loss': 0.0, 'grad_norm': 0.0008447727424018299, 'learning_rate': 5.583774250440917e-06, 'epoch': 1.0}\n",
      "{'loss': 0.0, 'grad_norm': 0.0003243405900706901, 'learning_rate': 5.580246913580247e-06, 'epoch': 1.0}\n",
      "{'loss': 0.0, 'grad_norm': 8.97245429360491e-05, 'learning_rate': 5.576719576719577e-06, 'epoch': 1.0}\n",
      "{'loss': 0.0, 'grad_norm': 9.4976878092401e-05, 'learning_rate': 5.573192239858906e-06, 'epoch': 1.0}\n",
      "{'loss': 0.0001, 'grad_norm': 0.005478179049490882, 'learning_rate': 5.569664902998236e-06, 'epoch': 1.0}\n",
      "{'loss': 0.0044, 'grad_norm': 0.5069476465736646, 'learning_rate': 5.566137566137566e-06, 'epoch': 1.0}\n",
      "{'loss': 0.1853, 'grad_norm': 6.898590120763581, 'learning_rate': 5.562610229276897e-06, 'epoch': 1.0}\n",
      "{'loss': 0.0006, 'grad_norm': 0.05876191366224594, 'learning_rate': 5.559082892416227e-06, 'epoch': 1.0}\n",
      "{'loss': 0.0, 'grad_norm': 0.0005429537819776038, 'learning_rate': 5.555555555555557e-06, 'epoch': 1.0}\n",
      "{'loss': 0.0005, 'grad_norm': 0.08539718405983514, 'learning_rate': 5.5520282186948855e-06, 'epoch': 1.0}\n",
      "{'loss': 0.0, 'grad_norm': 0.0006038824693148347, 'learning_rate': 5.548500881834215e-06, 'epoch': 1.0}\n",
      "{'loss': 0.0015, 'grad_norm': 0.16320288342667555, 'learning_rate': 5.544973544973545e-06, 'epoch': 1.0}\n",
      "{'loss': 0.0002, 'grad_norm': 0.015672844227543045, 'learning_rate': 5.541446208112875e-06, 'epoch': 1.0}\n",
      "{'loss': 0.0001, 'grad_norm': 0.020426847224434913, 'learning_rate': 5.537918871252204e-06, 'epoch': 1.01}\n",
      "{'loss': 0.0003, 'grad_norm': 0.03786827975782086, 'learning_rate': 5.534391534391534e-06, 'epoch': 1.01}\n",
      "{'loss': 0.0024, 'grad_norm': 0.20076012370365398, 'learning_rate': 5.530864197530864e-06, 'epoch': 1.01}\n",
      "{'loss': 0.0141, 'grad_norm': 1.4330341795780264, 'learning_rate': 5.527336860670195e-06, 'epoch': 1.01}\n",
      "{'loss': 0.0, 'grad_norm': 2.1406633149362134e-05, 'learning_rate': 5.523809523809525e-06, 'epoch': 1.01}\n",
      "{'loss': 0.0005, 'grad_norm': 0.045026972397766866, 'learning_rate': 5.520282186948855e-06, 'epoch': 1.01}\n",
      "{'loss': 0.1245, 'grad_norm': 7.080633598899553, 'learning_rate': 5.516754850088184e-06, 'epoch': 1.01}\n",
      "{'loss': 0.0, 'grad_norm': 0.0054535431971471015, 'learning_rate': 5.513227513227514e-06, 'epoch': 1.01}\n",
      "{'loss': 0.0003, 'grad_norm': 0.03144694088634972, 'learning_rate': 5.509700176366844e-06, 'epoch': 1.01}\n",
      "{'loss': 0.0001, 'grad_norm': 0.015659334598592197, 'learning_rate': 5.5061728395061735e-06, 'epoch': 1.01}\n",
      "{'loss': 0.0001, 'grad_norm': 0.009063113397745086, 'learning_rate': 5.502645502645503e-06, 'epoch': 1.01}\n",
      "{'loss': 0.3005, 'grad_norm': 20.292413445843035, 'learning_rate': 5.499118165784832e-06, 'epoch': 1.01}\n",
      "{'loss': 0.0, 'grad_norm': 0.00019271049065540714, 'learning_rate': 5.495590828924162e-06, 'epoch': 1.01}\n",
      "{'loss': 0.002, 'grad_norm': 0.21997880656127425, 'learning_rate': 5.492063492063493e-06, 'epoch': 1.01}\n",
      "{'loss': 0.0, 'grad_norm': 0.00024310954728549384, 'learning_rate': 5.488536155202823e-06, 'epoch': 1.01}\n",
      "{'loss': 0.0, 'grad_norm': 0.0009597425222629731, 'learning_rate': 5.485008818342153e-06, 'epoch': 1.01}\n",
      "{'loss': 0.0001, 'grad_norm': 0.00781334466391392, 'learning_rate': 5.481481481481482e-06, 'epoch': 1.02}\n",
      "{'loss': 0.0, 'grad_norm': 4.074988105958929e-05, 'learning_rate': 5.477954144620812e-06, 'epoch': 1.02}\n",
      "{'loss': 0.0, 'grad_norm': 0.0006309431140359905, 'learning_rate': 5.474426807760142e-06, 'epoch': 1.02}\n",
      "{'loss': 0.0, 'grad_norm': 0.00014643570579540662, 'learning_rate': 5.4708994708994715e-06, 'epoch': 1.02}\n",
      "{'loss': 0.0002, 'grad_norm': 0.016642747310507747, 'learning_rate': 5.467372134038801e-06, 'epoch': 1.02}\n",
      "{'loss': 0.0, 'grad_norm': 1.4639376099042506e-05, 'learning_rate': 5.463844797178131e-06, 'epoch': 1.02}\n",
      "{'loss': 1.3682, 'grad_norm': 27.080554745287255, 'learning_rate': 5.460317460317461e-06, 'epoch': 1.02}\n",
      "{'loss': 0.0, 'grad_norm': 0.000937862901044928, 'learning_rate': 5.456790123456791e-06, 'epoch': 1.02}\n",
      "{'loss': 0.074, 'grad_norm': 8.45244128834446, 'learning_rate': 5.453262786596121e-06, 'epoch': 1.02}\n",
      "{'loss': 0.0, 'grad_norm': 0.00042128443178072534, 'learning_rate': 5.449735449735451e-06, 'epoch': 1.02}\n",
      "{'loss': 0.0003, 'grad_norm': 0.03314935488659764, 'learning_rate': 5.44620811287478e-06, 'epoch': 1.02}\n",
      "{'loss': 0.0019, 'grad_norm': 0.15335537576343442, 'learning_rate': 5.44268077601411e-06, 'epoch': 1.02}\n",
      "{'loss': 0.729, 'grad_norm': 27.03456672209789, 'learning_rate': 5.43915343915344e-06, 'epoch': 1.02}\n",
      "{'loss': 0.2499, 'grad_norm': 10.058126506138038, 'learning_rate': 5.4356261022927694e-06, 'epoch': 1.02}\n",
      "{'loss': 0.0003, 'grad_norm': 0.032331008428544604, 'learning_rate': 5.432098765432099e-06, 'epoch': 1.02}\n",
      "{'loss': 0.0013, 'grad_norm': 0.17708685581593367, 'learning_rate': 5.428571428571429e-06, 'epoch': 1.02}\n",
      "{'loss': 0.0, 'grad_norm': 1.024904037468972e-05, 'learning_rate': 5.425044091710759e-06, 'epoch': 1.03}\n",
      "{'loss': 0.0184, 'grad_norm': 1.649667651488269, 'learning_rate': 5.421516754850089e-06, 'epoch': 1.03}\n",
      "{'loss': 0.0, 'grad_norm': 0.0009565809777155576, 'learning_rate': 5.417989417989419e-06, 'epoch': 1.03}\n",
      "{'loss': 0.0, 'grad_norm': 0.0003772036042616874, 'learning_rate': 5.4144620811287486e-06, 'epoch': 1.03}\n",
      "{'loss': 0.0, 'grad_norm': 0.0019131538143901216, 'learning_rate': 5.410934744268078e-06, 'epoch': 1.03}\n",
      "{'loss': 0.0, 'grad_norm': 0.0040838443304312575, 'learning_rate': 5.407407407407408e-06, 'epoch': 1.03}\n",
      "{'loss': 0.0, 'grad_norm': 5.0024503127522264e-05, 'learning_rate': 5.403880070546738e-06, 'epoch': 1.03}\n",
      "{'loss': 0.0, 'grad_norm': 5.666070092165322e-05, 'learning_rate': 5.400352733686067e-06, 'epoch': 1.03}\n",
      "{'loss': 0.0, 'grad_norm': 0.0006222361795200565, 'learning_rate': 5.396825396825397e-06, 'epoch': 1.03}\n",
      "{'loss': 0.0, 'grad_norm': 0.0007999247830844412, 'learning_rate': 5.393298059964727e-06, 'epoch': 1.03}\n",
      "{'loss': 0.0064, 'grad_norm': 0.7821236112855712, 'learning_rate': 5.3897707231040566e-06, 'epoch': 1.03}\n",
      "{'loss': 0.0, 'grad_norm': 0.0021066659173580226, 'learning_rate': 5.386243386243387e-06, 'epoch': 1.03}\n",
      "{'loss': 0.0002, 'grad_norm': 0.029819034437782575, 'learning_rate': 5.382716049382717e-06, 'epoch': 1.03}\n",
      "{'loss': 0.0001, 'grad_norm': 0.01323251400894381, 'learning_rate': 5.3791887125220465e-06, 'epoch': 1.03}\n",
      "{'loss': 0.0, 'grad_norm': 3.576937846939388e-05, 'learning_rate': 5.375661375661376e-06, 'epoch': 1.03}\n",
      "{'loss': 0.0, 'grad_norm': 0.0020000208020583645, 'learning_rate': 5.372134038800706e-06, 'epoch': 1.03}\n",
      "{'loss': 0.0005, 'grad_norm': 0.05385570523387252, 'learning_rate': 5.368606701940036e-06, 'epoch': 1.04}\n",
      "{'loss': 0.0, 'grad_norm': 1.6156075085239376e-05, 'learning_rate': 5.365079365079365e-06, 'epoch': 1.04}\n",
      "{'loss': 0.1129, 'grad_norm': 9.420129819975694, 'learning_rate': 5.361552028218695e-06, 'epoch': 1.04}\n",
      "{'loss': 0.0001, 'grad_norm': 0.016363013969889838, 'learning_rate': 5.358024691358025e-06, 'epoch': 1.04}\n",
      "{'loss': 0.0001, 'grad_norm': 0.008397505640573469, 'learning_rate': 5.3544973544973545e-06, 'epoch': 1.04}\n",
      "{'loss': 0.0, 'grad_norm': 0.000248784517923328, 'learning_rate': 5.350970017636685e-06, 'epoch': 1.04}\n",
      "{'loss': 0.0002, 'grad_norm': 0.020841057998101216, 'learning_rate': 5.347442680776015e-06, 'epoch': 1.04}\n",
      "{'loss': 0.0011, 'grad_norm': 0.15792794490119835, 'learning_rate': 5.3439153439153445e-06, 'epoch': 1.04}\n",
      "{'loss': 0.0046, 'grad_norm': 0.6320459880454377, 'learning_rate': 5.340388007054674e-06, 'epoch': 1.04}\n",
      "{'loss': 0.4912, 'grad_norm': 23.691677721548526, 'learning_rate': 5.336860670194004e-06, 'epoch': 1.04}\n",
      "{'loss': 0.0, 'grad_norm': 0.00045659014474855005, 'learning_rate': 5.333333333333334e-06, 'epoch': 1.04}\n",
      "{'loss': 0.0022, 'grad_norm': 0.26744190284829433, 'learning_rate': 5.329805996472663e-06, 'epoch': 1.04}\n",
      "{'loss': 0.0, 'grad_norm': 3.1013106060211957e-06, 'learning_rate': 5.326278659611993e-06, 'epoch': 1.04}\n",
      "{'loss': 0.0046, 'grad_norm': 0.5255658166068279, 'learning_rate': 5.322751322751323e-06, 'epoch': 1.04}\n",
      "{'loss': 0.0, 'grad_norm': 0.003123495662551566, 'learning_rate': 5.3192239858906525e-06, 'epoch': 1.04}\n",
      "{'loss': 0.0, 'grad_norm': 0.001076539169762835, 'learning_rate': 5.315696649029983e-06, 'epoch': 1.05}\n",
      "{'loss': 0.0001, 'grad_norm': 0.01096306928301408, 'learning_rate': 5.312169312169313e-06, 'epoch': 1.05}\n",
      "{'loss': 0.0014, 'grad_norm': 0.13974273272403875, 'learning_rate': 5.3086419753086425e-06, 'epoch': 1.05}\n",
      "{'loss': 0.0, 'grad_norm': 0.0006226822127223948, 'learning_rate': 5.305114638447972e-06, 'epoch': 1.05}\n",
      "{'loss': 0.1025, 'grad_norm': 7.461874378759319, 'learning_rate': 5.301587301587302e-06, 'epoch': 1.05}\n",
      "{'loss': 0.0014, 'grad_norm': 0.13872060456854848, 'learning_rate': 5.298059964726632e-06, 'epoch': 1.05}\n",
      "{'loss': 0.0, 'grad_norm': 0.002010231788469045, 'learning_rate': 5.294532627865961e-06, 'epoch': 1.05}\n",
      "{'loss': 0.0, 'grad_norm': 0.0004998994901020252, 'learning_rate': 5.291005291005291e-06, 'epoch': 1.05}\n",
      "{'loss': 0.0005, 'grad_norm': 0.049836559942998626, 'learning_rate': 5.287477954144621e-06, 'epoch': 1.05}\n",
      "{'loss': 0.0028, 'grad_norm': 0.23778096037511273, 'learning_rate': 5.2839506172839505e-06, 'epoch': 1.05}\n",
      "{'loss': 0.0345, 'grad_norm': 4.044060276868931, 'learning_rate': 5.280423280423281e-06, 'epoch': 1.05}\n",
      "{'loss': 0.0, 'grad_norm': 9.755314788040322e-06, 'learning_rate': 5.276895943562611e-06, 'epoch': 1.05}\n",
      "{'loss': 0.0001, 'grad_norm': 0.007459366173261044, 'learning_rate': 5.2733686067019405e-06, 'epoch': 1.05}\n",
      "{'loss': 0.0019, 'grad_norm': 0.24541725546135296, 'learning_rate': 5.26984126984127e-06, 'epoch': 1.05}\n",
      "{'loss': 0.0018, 'grad_norm': 0.22437740454806546, 'learning_rate': 5.2663139329806e-06, 'epoch': 1.05}\n",
      "{'loss': 0.0, 'grad_norm': 0.0006425520483135045, 'learning_rate': 5.26278659611993e-06, 'epoch': 1.05}\n",
      "{'loss': 0.0, 'grad_norm': 3.805246166994784e-05, 'learning_rate': 5.259259259259259e-06, 'epoch': 1.06}\n",
      "{'loss': 0.0011, 'grad_norm': 0.15697231823812546, 'learning_rate': 5.255731922398589e-06, 'epoch': 1.06}\n",
      "{'loss': 0.0006, 'grad_norm': 0.08707649878563374, 'learning_rate': 5.252204585537919e-06, 'epoch': 1.06}\n",
      "{'loss': 0.002, 'grad_norm': 0.21514196506359667, 'learning_rate': 5.2486772486772485e-06, 'epoch': 1.06}\n",
      "{'loss': 0.2339, 'grad_norm': 16.33670061136074, 'learning_rate': 5.245149911816579e-06, 'epoch': 1.06}\n",
      "{'loss': 0.0, 'grad_norm': 0.0005510610555651715, 'learning_rate': 5.241622574955909e-06, 'epoch': 1.06}\n",
      "{'loss': 0.0, 'grad_norm': 3.454706241396886e-05, 'learning_rate': 5.2380952380952384e-06, 'epoch': 1.06}\n",
      "{'loss': 0.002, 'grad_norm': 0.2414657312710274, 'learning_rate': 5.234567901234568e-06, 'epoch': 1.06}\n",
      "{'loss': 0.0649, 'grad_norm': 7.812926257902719, 'learning_rate': 5.231040564373898e-06, 'epoch': 1.06}\n",
      "{'loss': 0.0, 'grad_norm': 0.001111964367497323, 'learning_rate': 5.227513227513228e-06, 'epoch': 1.06}\n",
      "{'loss': 0.0, 'grad_norm': 0.0031023195549457583, 'learning_rate': 5.223985890652557e-06, 'epoch': 1.06}\n",
      "{'loss': 0.0, 'grad_norm': 0.00018374945180971316, 'learning_rate': 5.220458553791887e-06, 'epoch': 1.06}\n",
      "{'loss': 0.0001, 'grad_norm': 0.010636735593891536, 'learning_rate': 5.216931216931217e-06, 'epoch': 1.06}\n",
      "{'loss': 0.0, 'grad_norm': 0.0004472560176446026, 'learning_rate': 5.2134038800705464e-06, 'epoch': 1.06}\n",
      "{'loss': 0.0018, 'grad_norm': 0.21234232613631412, 'learning_rate': 5.209876543209878e-06, 'epoch': 1.06}\n",
      "{'loss': 0.0, 'grad_norm': 0.00015406636783498183, 'learning_rate': 5.2063492063492076e-06, 'epoch': 1.06}\n",
      "{'loss': 0.0001, 'grad_norm': 0.013710297405042704, 'learning_rate': 5.202821869488537e-06, 'epoch': 1.07}\n",
      "{'loss': 0.0, 'grad_norm': 0.00013566404077973703, 'learning_rate': 5.199294532627866e-06, 'epoch': 1.07}\n",
      "{'loss': 0.0001, 'grad_norm': 0.016366128063566823, 'learning_rate': 5.195767195767196e-06, 'epoch': 1.07}\n",
      "{'loss': 0.0, 'grad_norm': 1.8976775772993566e-05, 'learning_rate': 5.1922398589065256e-06, 'epoch': 1.07}\n",
      "{'loss': 0.0137, 'grad_norm': 1.254929407345672, 'learning_rate': 5.188712522045855e-06, 'epoch': 1.07}\n",
      "{'loss': 0.0, 'grad_norm': 0.0025407625330554433, 'learning_rate': 5.185185185185185e-06, 'epoch': 1.07}\n",
      "{'loss': 0.0002, 'grad_norm': 0.023543393060912295, 'learning_rate': 5.181657848324515e-06, 'epoch': 1.07}\n",
      "{'loss': 0.0, 'grad_norm': 0.0008292574172026155, 'learning_rate': 5.178130511463844e-06, 'epoch': 1.07}\n",
      "{'loss': 0.0006, 'grad_norm': 0.07903146875972647, 'learning_rate': 5.174603174603176e-06, 'epoch': 1.07}\n",
      "{'loss': 0.0, 'grad_norm': 0.000753041751320378, 'learning_rate': 5.1710758377425055e-06, 'epoch': 1.07}\n",
      "{'loss': 0.0001, 'grad_norm': 0.010258778127127026, 'learning_rate': 5.167548500881835e-06, 'epoch': 1.07}\n",
      "{'loss': 0.1853, 'grad_norm': 13.565869172171917, 'learning_rate': 5.164021164021165e-06, 'epoch': 1.07}\n",
      "{'loss': 0.0, 'grad_norm': 0.0004378359546349093, 'learning_rate': 5.160493827160495e-06, 'epoch': 1.07}\n",
      "{'loss': 0.0001, 'grad_norm': 0.006841864794164109, 'learning_rate': 5.156966490299824e-06, 'epoch': 1.07}\n",
      "{'loss': 0.0, 'grad_norm': 6.479153520990214e-06, 'learning_rate': 5.153439153439154e-06, 'epoch': 1.07}\n",
      "{'loss': 0.0, 'grad_norm': 3.140999540666604e-05, 'learning_rate': 5.149911816578484e-06, 'epoch': 1.07}\n",
      "{'loss': 0.0333, 'grad_norm': 3.751887291604236, 'learning_rate': 5.146384479717813e-06, 'epoch': 1.08}\n",
      "{'loss': 0.0, 'grad_norm': 2.074464487957872e-05, 'learning_rate': 5.142857142857142e-06, 'epoch': 1.08}\n",
      "{'loss': 0.0, 'grad_norm': 8.779737367446641e-05, 'learning_rate': 5.139329805996474e-06, 'epoch': 1.08}\n",
      "{'loss': 0.0, 'grad_norm': 8.83672479999859e-05, 'learning_rate': 5.1358024691358035e-06, 'epoch': 1.08}\n",
      "{'loss': 0.0011, 'grad_norm': 0.1346776025215606, 'learning_rate': 5.132275132275133e-06, 'epoch': 1.08}\n",
      "{'loss': 0.0265, 'grad_norm': 1.760680440369048, 'learning_rate': 5.128747795414463e-06, 'epoch': 1.08}\n",
      "{'loss': 0.0008, 'grad_norm': 0.08009542592330972, 'learning_rate': 5.125220458553793e-06, 'epoch': 1.08}\n",
      "{'loss': 0.0, 'grad_norm': 0.00025620955158579665, 'learning_rate': 5.121693121693122e-06, 'epoch': 1.08}\n",
      "{'loss': 0.0233, 'grad_norm': 2.197472104013769, 'learning_rate': 5.118165784832452e-06, 'epoch': 1.08}\n",
      "{'loss': 0.0, 'grad_norm': 0.001474019119814793, 'learning_rate': 5.114638447971782e-06, 'epoch': 1.08}\n",
      "{'loss': 1.0273, 'grad_norm': 23.49261878137538, 'learning_rate': 5.1111111111111115e-06, 'epoch': 1.08}\n",
      "{'loss': 0.0, 'grad_norm': 3.118896335176573e-05, 'learning_rate': 5.107583774250441e-06, 'epoch': 1.08}\n",
      "{'loss': 0.0001, 'grad_norm': 0.009191255987387445, 'learning_rate': 5.104056437389772e-06, 'epoch': 1.08}\n",
      "{'loss': 0.0, 'grad_norm': 0.0007813534485371006, 'learning_rate': 5.1005291005291015e-06, 'epoch': 1.08}\n",
      "{'loss': 0.0005, 'grad_norm': 0.07597356437318054, 'learning_rate': 5.097001763668431e-06, 'epoch': 1.08}\n",
      "{'loss': 0.0, 'grad_norm': 0.002391200998227858, 'learning_rate': 5.093474426807761e-06, 'epoch': 1.09}\n",
      "{'loss': 0.0015, 'grad_norm': 0.1704883723849244, 'learning_rate': 5.089947089947091e-06, 'epoch': 1.09}\n",
      "{'loss': 0.0, 'grad_norm': 0.0010759719796924832, 'learning_rate': 5.08641975308642e-06, 'epoch': 1.09}\n",
      "{'loss': 0.0003, 'grad_norm': 0.03608214847297454, 'learning_rate': 5.08289241622575e-06, 'epoch': 1.09}\n",
      "{'loss': 0.0, 'grad_norm': 3.1149160462523724e-05, 'learning_rate': 5.07936507936508e-06, 'epoch': 1.09}\n",
      "{'loss': 0.0, 'grad_norm': 1.2399904500714452e-05, 'learning_rate': 5.0758377425044095e-06, 'epoch': 1.09}\n",
      "{'loss': 0.0453, 'grad_norm': 10.813210524658526, 'learning_rate': 5.072310405643739e-06, 'epoch': 1.09}\n",
      "{'loss': 0.5605, 'grad_norm': 29.971978897679893, 'learning_rate': 5.06878306878307e-06, 'epoch': 1.09}\n",
      "{'loss': 0.0, 'grad_norm': 4.208805743410603e-05, 'learning_rate': 5.0652557319223995e-06, 'epoch': 1.09}\n",
      "{'loss': 0.0, 'grad_norm': 0.0009825452092981958, 'learning_rate': 5.061728395061729e-06, 'epoch': 1.09}\n",
      "{'loss': 0.0, 'grad_norm': 0.007587210382006612, 'learning_rate': 5.058201058201059e-06, 'epoch': 1.09}\n",
      "{'loss': 0.0003, 'grad_norm': 0.03594203957879763, 'learning_rate': 5.054673721340389e-06, 'epoch': 1.09}\n",
      "{'loss': 0.0016, 'grad_norm': 0.2256544347699824, 'learning_rate': 5.051146384479718e-06, 'epoch': 1.09}\n",
      "{'loss': 0.0, 'grad_norm': 0.00021319459132974238, 'learning_rate': 5.047619047619048e-06, 'epoch': 1.09}\n",
      "{'loss': 0.0, 'grad_norm': 0.00026146800170393235, 'learning_rate': 5.044091710758378e-06, 'epoch': 1.09}\n",
      "{'loss': 0.0, 'grad_norm': 0.0011064254397785212, 'learning_rate': 5.0405643738977074e-06, 'epoch': 1.09}\n",
      "{'loss': 0.0001, 'grad_norm': 0.008689927677899906, 'learning_rate': 5.037037037037037e-06, 'epoch': 1.1}\n",
      "{'loss': 0.0002, 'grad_norm': 0.020713008773939758, 'learning_rate': 5.033509700176368e-06, 'epoch': 1.1}\n",
      "{'loss': 0.0, 'grad_norm': 0.002552208516616236, 'learning_rate': 5.0299823633156974e-06, 'epoch': 1.1}\n",
      "{'loss': 0.0, 'grad_norm': 0.004388458970745266, 'learning_rate': 5.026455026455027e-06, 'epoch': 1.1}\n",
      "{'loss': 0.0004, 'grad_norm': 0.030615896160024504, 'learning_rate': 5.022927689594357e-06, 'epoch': 1.1}\n",
      "{'loss': 0.0, 'grad_norm': 0.00012251334197160353, 'learning_rate': 5.0194003527336866e-06, 'epoch': 1.1}\n",
      "{'loss': 1.1514, 'grad_norm': 22.618810660667485, 'learning_rate': 5.015873015873016e-06, 'epoch': 1.1}\n",
      "{'loss': 0.0, 'grad_norm': 0.00018876815467050136, 'learning_rate': 5.012345679012346e-06, 'epoch': 1.1}\n",
      "{'loss': 0.0001, 'grad_norm': 0.015513836316942456, 'learning_rate': 5.008818342151676e-06, 'epoch': 1.1}\n",
      "{'loss': 0.0, 'grad_norm': 0.003123872715917862, 'learning_rate': 5.005291005291005e-06, 'epoch': 1.1}\n",
      "{'loss': 0.0, 'grad_norm': 0.004935692939044747, 'learning_rate': 5.001763668430335e-06, 'epoch': 1.1}\n",
      "{'loss': 0.0, 'grad_norm': 0.0005240408912057505, 'learning_rate': 4.998236331569665e-06, 'epoch': 1.1}\n",
      "{'loss': 0.2174, 'grad_norm': 16.4392200338783, 'learning_rate': 4.9947089947089946e-06, 'epoch': 1.1}\n",
      "{'loss': 0.0, 'grad_norm': 4.146586979303532e-06, 'learning_rate': 4.991181657848324e-06, 'epoch': 1.1}\n",
      "{'loss': 0.0, 'grad_norm': 1.973422175864017e-05, 'learning_rate': 4.987654320987655e-06, 'epoch': 1.1}\n",
      "{'loss': 0.0003, 'grad_norm': 0.03073219555303868, 'learning_rate': 4.9841269841269845e-06, 'epoch': 1.1}\n",
      "{'loss': 0.0366, 'grad_norm': 2.5219144683358725, 'learning_rate': 4.980599647266314e-06, 'epoch': 1.11}\n",
      "{'loss': 0.0, 'grad_norm': 0.0001325485217223993, 'learning_rate': 4.977072310405644e-06, 'epoch': 1.11}\n",
      "{'loss': 0.0, 'grad_norm': 4.7242246263864396e-05, 'learning_rate': 4.973544973544974e-06, 'epoch': 1.11}\n",
      "{'loss': 0.0, 'grad_norm': 0.00013868274963509822, 'learning_rate': 4.970017636684304e-06, 'epoch': 1.11}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 55%|█████▌    | 1745/3150 [07:21<04:42,  4.97it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.0009, 'grad_norm': 0.09674130508777008, 'learning_rate': 4.966490299823634e-06, 'epoch': 1.11}\n",
      "{'loss': 0.0, 'grad_norm': 0.0059433610812439225, 'learning_rate': 4.962962962962964e-06, 'epoch': 1.11}\n",
      "{'loss': 0.0, 'grad_norm': 0.0023787512831038777, 'learning_rate': 4.959435626102293e-06, 'epoch': 1.11}\n",
      "{'loss': 0.0001, 'grad_norm': 0.0053428588825208895, 'learning_rate': 4.955908289241623e-06, 'epoch': 1.11}\n",
      "{'loss': 0.0006, 'grad_norm': 0.06979952889194821, 'learning_rate': 4.952380952380953e-06, 'epoch': 1.11}\n",
      "{'loss': 0.0015, 'grad_norm': 0.12311688959209235, 'learning_rate': 4.9488536155202825e-06, 'epoch': 1.11}\n",
      "{'loss': 0.0, 'grad_norm': 0.005659354743994171, 'learning_rate': 4.945326278659612e-06, 'epoch': 1.11}\n",
      "{'loss': 0.1051, 'grad_norm': 12.496530280160519, 'learning_rate': 4.941798941798942e-06, 'epoch': 1.11}\n",
      "{'loss': 0.0, 'grad_norm': 7.150547607013999e-05, 'learning_rate': 4.938271604938272e-06, 'epoch': 1.11}\n",
      "{'loss': 0.0013, 'grad_norm': 0.11405076724287291, 'learning_rate': 4.934744268077602e-06, 'epoch': 1.11}\n",
      "{'loss': 0.0, 'grad_norm': 0.0030610510767908176, 'learning_rate': 4.931216931216932e-06, 'epoch': 1.11}\n",
      "{'loss': 0.0, 'grad_norm': 0.0007240882152970541, 'learning_rate': 4.927689594356262e-06, 'epoch': 1.11}\n",
      "{'loss': 0.0003, 'grad_norm': 0.03784171842751524, 'learning_rate': 4.924162257495591e-06, 'epoch': 1.12}\n",
      "{'loss': 0.0, 'grad_norm': 0.00038246135793825207, 'learning_rate': 4.920634920634921e-06, 'epoch': 1.12}\n",
      "{'loss': 0.0, 'grad_norm': 0.00028488683305452573, 'learning_rate': 4.917107583774251e-06, 'epoch': 1.12}\n",
      "{'loss': 0.0318, 'grad_norm': 4.74978396275779, 'learning_rate': 4.9135802469135805e-06, 'epoch': 1.12}\n",
      "{'loss': 0.0, 'grad_norm': 0.0013182847774006136, 'learning_rate': 4.91005291005291e-06, 'epoch': 1.12}\n",
      "{'loss': 0.0, 'grad_norm': 0.0003757547151565451, 'learning_rate': 4.90652557319224e-06, 'epoch': 1.12}\n",
      "{'loss': 0.0, 'grad_norm': 0.003282077330127417, 'learning_rate': 4.90299823633157e-06, 'epoch': 1.12}\n",
      "{'loss': 0.312, 'grad_norm': 24.00191744456315, 'learning_rate': 4.8994708994709e-06, 'epoch': 1.12}\n",
      "{'loss': 0.0, 'grad_norm': 8.998080715432804e-05, 'learning_rate': 4.89594356261023e-06, 'epoch': 1.12}\n",
      "{'loss': 0.0, 'grad_norm': 7.10073494186763e-05, 'learning_rate': 4.89241622574956e-06, 'epoch': 1.12}\n",
      "{'loss': 0.0, 'grad_norm': 2.5280527025015438e-05, 'learning_rate': 4.888888888888889e-06, 'epoch': 1.12}\n",
      "{'loss': 0.0, 'grad_norm': 8.404484851086188e-06, 'learning_rate': 4.885361552028219e-06, 'epoch': 1.12}\n",
      "{'loss': 0.0001, 'grad_norm': 0.008575563328075663, 'learning_rate': 4.881834215167549e-06, 'epoch': 1.12}\n",
      "{'loss': 0.0, 'grad_norm': 0.0005834935623396097, 'learning_rate': 4.8783068783068785e-06, 'epoch': 1.12}\n",
      "{'loss': 0.0, 'grad_norm': 0.0017185743050595502, 'learning_rate': 4.874779541446208e-06, 'epoch': 1.12}\n",
      "{'loss': 0.0005, 'grad_norm': 0.0761488763617114, 'learning_rate': 4.871252204585538e-06, 'epoch': 1.13}\n",
      "{'loss': 0.0005, 'grad_norm': 0.061732955320940815, 'learning_rate': 4.867724867724868e-06, 'epoch': 1.13}\n",
      "{'loss': 0.0001, 'grad_norm': 0.017099944603758622, 'learning_rate': 4.864197530864198e-06, 'epoch': 1.13}\n",
      "{'loss': 0.0127, 'grad_norm': 1.4618768782846192, 'learning_rate': 4.860670194003528e-06, 'epoch': 1.13}\n",
      "{'loss': 0.0003, 'grad_norm': 0.02714642677648208, 'learning_rate': 4.857142857142858e-06, 'epoch': 1.13}\n",
      "{'loss': 0.0, 'grad_norm': 0.00017837780177786357, 'learning_rate': 4.853615520282187e-06, 'epoch': 1.13}\n",
      "{'loss': 0.0, 'grad_norm': 0.0020431925277778398, 'learning_rate': 4.850088183421517e-06, 'epoch': 1.13}\n",
      "{'loss': 0.0087, 'grad_norm': 0.7388935678337097, 'learning_rate': 4.846560846560847e-06, 'epoch': 1.13}\n",
      "{'loss': 0.0004, 'grad_norm': 0.04026840978702523, 'learning_rate': 4.8430335097001764e-06, 'epoch': 1.13}\n",
      "{'loss': 0.0, 'grad_norm': 0.0014911974988560108, 'learning_rate': 4.839506172839506e-06, 'epoch': 1.13}\n",
      "{'loss': 0.0005, 'grad_norm': 0.0535164171703209, 'learning_rate': 4.835978835978836e-06, 'epoch': 1.13}\n",
      "{'loss': 0.0027, 'grad_norm': 0.4621642808066184, 'learning_rate': 4.832451499118166e-06, 'epoch': 1.13}\n",
      "{'loss': 0.0001, 'grad_norm': 0.006760560838878801, 'learning_rate': 4.828924162257496e-06, 'epoch': 1.13}\n",
      "{'loss': 0.0001, 'grad_norm': 0.006955237474978831, 'learning_rate': 4.825396825396826e-06, 'epoch': 1.13}\n",
      "{'loss': 0.0, 'grad_norm': 9.712210117026762e-06, 'learning_rate': 4.8218694885361556e-06, 'epoch': 1.13}\n",
      "{'loss': 0.0001, 'grad_norm': 0.009627079546794466, 'learning_rate': 4.818342151675485e-06, 'epoch': 1.13}\n",
      "{'loss': 0.0039, 'grad_norm': 0.5180472209516788, 'learning_rate': 4.814814814814815e-06, 'epoch': 1.14}\n",
      "{'loss': 0.0, 'grad_norm': 6.474812946112351e-05, 'learning_rate': 4.8112874779541455e-06, 'epoch': 1.14}\n",
      "{'loss': 0.0001, 'grad_norm': 0.01953610419509878, 'learning_rate': 4.807760141093475e-06, 'epoch': 1.14}\n",
      "{'loss': 0.0, 'grad_norm': 2.846890726206697e-06, 'learning_rate': 4.804232804232805e-06, 'epoch': 1.14}\n",
      "{'loss': 0.0, 'grad_norm': 0.0014600446219415948, 'learning_rate': 4.800705467372135e-06, 'epoch': 1.14}\n",
      "{'loss': 0.0143, 'grad_norm': 1.2038413058122623, 'learning_rate': 4.7971781305114636e-06, 'epoch': 1.14}\n",
      "{'loss': 0.1907, 'grad_norm': 15.426466732680305, 'learning_rate': 4.793650793650794e-06, 'epoch': 1.14}\n",
      "{'loss': 0.0, 'grad_norm': 0.00010844705583043408, 'learning_rate': 4.790123456790124e-06, 'epoch': 1.14}\n",
      "{'loss': 0.0, 'grad_norm': 8.891671095621804e-05, 'learning_rate': 4.7865961199294535e-06, 'epoch': 1.14}\n",
      "{'loss': 0.0, 'grad_norm': 0.00010912215063822713, 'learning_rate': 4.783068783068783e-06, 'epoch': 1.14}\n",
      "{'loss': 0.0, 'grad_norm': 0.00022087627677679737, 'learning_rate': 4.779541446208113e-06, 'epoch': 1.14}\n",
      "{'loss': 0.0, 'grad_norm': 0.003237996068905928, 'learning_rate': 4.7760141093474435e-06, 'epoch': 1.14}\n",
      "{'loss': 0.0, 'grad_norm': 0.0033716568424497035, 'learning_rate': 4.772486772486773e-06, 'epoch': 1.14}\n",
      "{'loss': 0.0002, 'grad_norm': 0.023882208286799617, 'learning_rate': 4.768959435626103e-06, 'epoch': 1.14}\n",
      "{'loss': 0.052, 'grad_norm': 3.013007099952271, 'learning_rate': 4.765432098765433e-06, 'epoch': 1.14}\n",
      "{'loss': 0.0001, 'grad_norm': 0.007072863057416513, 'learning_rate': 4.761904761904762e-06, 'epoch': 1.14}\n",
      "{'loss': 0.0, 'grad_norm': 0.0012592535835318748, 'learning_rate': 4.758377425044092e-06, 'epoch': 1.15}\n",
      "{'loss': 0.058, 'grad_norm': 4.946124114198081, 'learning_rate': 4.754850088183422e-06, 'epoch': 1.15}\n",
      "{'loss': 0.0, 'grad_norm': 3.5156470691935725e-05, 'learning_rate': 4.7513227513227515e-06, 'epoch': 1.15}\n",
      "{'loss': 0.0018, 'grad_norm': 0.20260646548270458, 'learning_rate': 4.747795414462081e-06, 'epoch': 1.15}\n",
      "{'loss': 0.0, 'grad_norm': 9.203472367175885e-06, 'learning_rate': 4.744268077601411e-06, 'epoch': 1.15}\n",
      "{'loss': 0.0009, 'grad_norm': 0.11930783060977143, 'learning_rate': 4.7407407407407415e-06, 'epoch': 1.15}\n",
      "{'loss': 0.0282, 'grad_norm': 3.857156463377029, 'learning_rate': 4.737213403880071e-06, 'epoch': 1.15}\n",
      "{'loss': 0.0348, 'grad_norm': 3.1123489649639087, 'learning_rate': 4.733686067019401e-06, 'epoch': 1.15}\n",
      "{'loss': 0.0, 'grad_norm': 7.672189560865386e-05, 'learning_rate': 4.730158730158731e-06, 'epoch': 1.15}\n",
      "{'loss': 0.0004, 'grad_norm': 0.02542276251349472, 'learning_rate': 4.72663139329806e-06, 'epoch': 1.15}\n",
      "{'loss': 0.3574, 'grad_norm': 11.650228709321212, 'learning_rate': 4.72310405643739e-06, 'epoch': 1.15}\n",
      "{'loss': 0.0001, 'grad_norm': 0.015947089040973434, 'learning_rate': 4.71957671957672e-06, 'epoch': 1.15}\n",
      "{'loss': 0.0, 'grad_norm': 0.002171086866524487, 'learning_rate': 4.7160493827160495e-06, 'epoch': 1.15}\n",
      "{'loss': 1.1123, 'grad_norm': 22.668068038404815, 'learning_rate': 4.712522045855379e-06, 'epoch': 1.15}\n",
      "{'loss': 0.0, 'grad_norm': 0.00015082786650412258, 'learning_rate': 4.708994708994709e-06, 'epoch': 1.15}\n",
      "{'loss': 0.0, 'grad_norm': 0.0035377210445568766, 'learning_rate': 4.7054673721340395e-06, 'epoch': 1.15}\n",
      "{'loss': 0.0001, 'grad_norm': 0.008619876774461264, 'learning_rate': 4.701940035273369e-06, 'epoch': 1.16}\n",
      "{'loss': 0.0, 'grad_norm': 0.0013669765358342617, 'learning_rate': 4.698412698412699e-06, 'epoch': 1.16}\n",
      "{'loss': 0.0006, 'grad_norm': 0.046138248543233826, 'learning_rate': 4.694885361552029e-06, 'epoch': 1.16}\n",
      "{'loss': 0.0, 'grad_norm': 0.001141179464629869, 'learning_rate': 4.691358024691358e-06, 'epoch': 1.16}\n",
      "{'loss': 0.0, 'grad_norm': 0.00010630627923450292, 'learning_rate': 4.687830687830688e-06, 'epoch': 1.16}\n",
      "{'loss': 0.0, 'grad_norm': 0.00021187785407739745, 'learning_rate': 4.684303350970018e-06, 'epoch': 1.16}\n",
      "{'loss': 0.0, 'grad_norm': 0.0002838840019658261, 'learning_rate': 4.6807760141093475e-06, 'epoch': 1.16}\n",
      "{'loss': 0.0186, 'grad_norm': 2.4341098978940194, 'learning_rate': 4.677248677248677e-06, 'epoch': 1.16}\n",
      "{'loss': 0.0, 'grad_norm': 0.0009650626597587486, 'learning_rate': 4.673721340388007e-06, 'epoch': 1.16}\n",
      "{'loss': 0.0047, 'grad_norm': 0.5702162883548825, 'learning_rate': 4.6701940035273374e-06, 'epoch': 1.16}\n",
      "{'loss': 0.0, 'grad_norm': 0.0007678663344853477, 'learning_rate': 4.666666666666667e-06, 'epoch': 1.16}\n",
      "{'loss': 0.0, 'grad_norm': 0.0008804668360481353, 'learning_rate': 4.663139329805997e-06, 'epoch': 1.16}\n",
      "{'loss': 0.0, 'grad_norm': 1.749328496539461e-05, 'learning_rate': 4.659611992945327e-06, 'epoch': 1.16}\n",
      "{'loss': 0.0829, 'grad_norm': 8.60399247578922, 'learning_rate': 4.656084656084656e-06, 'epoch': 1.16}\n",
      "{'loss': 0.0001, 'grad_norm': 0.014499751619890524, 'learning_rate': 4.652557319223987e-06, 'epoch': 1.16}\n",
      "{'loss': 0.0, 'grad_norm': 2.1074552970506107e-05, 'learning_rate': 4.6490299823633166e-06, 'epoch': 1.17}\n",
      "{'loss': 0.0617, 'grad_norm': 6.20134749997367, 'learning_rate': 4.6455026455026454e-06, 'epoch': 1.17}\n",
      "{'loss': 0.0, 'grad_norm': 0.0056447115145635786, 'learning_rate': 4.641975308641975e-06, 'epoch': 1.17}\n",
      "{'loss': 0.0, 'grad_norm': 0.0021083385011754427, 'learning_rate': 4.638447971781305e-06, 'epoch': 1.17}\n",
      "{'loss': 0.0, 'grad_norm': 4.772806573147897e-05, 'learning_rate': 4.634920634920635e-06, 'epoch': 1.17}\n",
      "{'loss': 0.0, 'grad_norm': 2.6043204735758855e-05, 'learning_rate': 4.631393298059965e-06, 'epoch': 1.17}\n",
      "{'loss': 0.0044, 'grad_norm': 0.34825655725156507, 'learning_rate': 4.627865961199295e-06, 'epoch': 1.17}\n",
      "{'loss': 0.0, 'grad_norm': 9.412478383218325e-05, 'learning_rate': 4.6243386243386246e-06, 'epoch': 1.17}\n",
      "{'loss': 0.0, 'grad_norm': 0.0014344513995715549, 'learning_rate': 4.620811287477954e-06, 'epoch': 1.17}\n",
      "{'loss': 0.0, 'grad_norm': 0.0012613078800787446, 'learning_rate': 4.617283950617285e-06, 'epoch': 1.17}\n",
      "{'loss': 0.0023, 'grad_norm': 0.28733033481332915, 'learning_rate': 4.6137566137566145e-06, 'epoch': 1.17}\n",
      "{'loss': 0.0, 'grad_norm': 0.0038425025854334164, 'learning_rate': 4.610229276895944e-06, 'epoch': 1.17}\n",
      "{'loss': 0.0, 'grad_norm': 0.0006413722343104416, 'learning_rate': 4.606701940035274e-06, 'epoch': 1.17}\n",
      "{'loss': 0.0, 'grad_norm': 0.003160235522962234, 'learning_rate': 4.603174603174604e-06, 'epoch': 1.17}\n",
      "{'loss': 0.0, 'grad_norm': 0.00013798485564357795, 'learning_rate': 4.599647266313933e-06, 'epoch': 1.17}\n",
      "{'loss': 0.0002, 'grad_norm': 0.02259460615330524, 'learning_rate': 4.596119929453263e-06, 'epoch': 1.17}\n",
      "{'loss': 0.0, 'grad_norm': 0.001039981514092437, 'learning_rate': 4.592592592592593e-06, 'epoch': 1.18}\n",
      "{'loss': 0.0, 'grad_norm': 5.510793519661592e-05, 'learning_rate': 4.5890652557319225e-06, 'epoch': 1.18}\n",
      "{'loss': 0.0, 'grad_norm': 0.0022156687911236645, 'learning_rate': 4.585537918871252e-06, 'epoch': 1.18}\n",
      "{'loss': 0.0, 'grad_norm': 0.0038308578546576363, 'learning_rate': 4.582010582010583e-06, 'epoch': 1.18}\n",
      "{'loss': 0.0001, 'grad_norm': 0.00654036826972274, 'learning_rate': 4.5784832451499125e-06, 'epoch': 1.18}\n",
      "{'loss': 0.0, 'grad_norm': 1.259618709043582e-05, 'learning_rate': 4.574955908289242e-06, 'epoch': 1.18}\n",
      "{'loss': 0.0, 'grad_norm': 5.870037458080145e-06, 'learning_rate': 4.571428571428572e-06, 'epoch': 1.18}\n",
      "{'loss': 0.0, 'grad_norm': 2.4880052930252676e-06, 'learning_rate': 4.567901234567902e-06, 'epoch': 1.18}\n",
      "{'loss': 0.0, 'grad_norm': 6.886980436151535e-05, 'learning_rate': 4.564373897707231e-06, 'epoch': 1.18}\n",
      "{'loss': 0.0, 'grad_norm': 1.2552208527502968e-05, 'learning_rate': 4.560846560846561e-06, 'epoch': 1.18}\n",
      "{'loss': 0.0017, 'grad_norm': 0.20296631630252643, 'learning_rate': 4.557319223985891e-06, 'epoch': 1.18}\n",
      "{'loss': 0.0, 'grad_norm': 3.5505213240897e-06, 'learning_rate': 4.5537918871252205e-06, 'epoch': 1.18}\n",
      "{'loss': 0.0018, 'grad_norm': 0.17434725234716691, 'learning_rate': 4.55026455026455e-06, 'epoch': 1.18}\n",
      "{'loss': 0.0, 'grad_norm': 0.004054777365811368, 'learning_rate': 4.546737213403881e-06, 'epoch': 1.18}\n",
      "{'loss': 0.0003, 'grad_norm': 0.034529238818685325, 'learning_rate': 4.5432098765432105e-06, 'epoch': 1.18}\n",
      "{'loss': 0.0, 'grad_norm': 0.0009872098351108019, 'learning_rate': 4.53968253968254e-06, 'epoch': 1.18}\n",
      "{'loss': 0.0, 'grad_norm': 7.791972977077685e-06, 'learning_rate': 4.53615520282187e-06, 'epoch': 1.19}\n",
      "{'loss': 0.0002, 'grad_norm': 0.02128623087232516, 'learning_rate': 4.5326278659612e-06, 'epoch': 1.19}\n",
      "{'loss': 0.0001, 'grad_norm': 0.011499070665541932, 'learning_rate': 4.529100529100529e-06, 'epoch': 1.19}\n",
      "{'loss': 0.0, 'grad_norm': 0.00030829888164148143, 'learning_rate': 4.525573192239859e-06, 'epoch': 1.19}\n",
      "{'loss': 0.0, 'grad_norm': 0.00025958535171482984, 'learning_rate': 4.522045855379189e-06, 'epoch': 1.19}\n",
      "{'loss': 0.0, 'grad_norm': 0.0012494756414843289, 'learning_rate': 4.5185185185185185e-06, 'epoch': 1.19}\n",
      "{'loss': 0.0, 'grad_norm': 0.0047130752603316, 'learning_rate': 4.514991181657848e-06, 'epoch': 1.19}\n",
      "{'loss': 0.0076, 'grad_norm': 0.8224319688437213, 'learning_rate': 4.511463844797179e-06, 'epoch': 1.19}\n",
      "{'loss': 0.0, 'grad_norm': 2.093372328170729e-06, 'learning_rate': 4.5079365079365085e-06, 'epoch': 1.19}\n",
      "{'loss': 0.0, 'grad_norm': 9.069784403807794e-05, 'learning_rate': 4.504409171075838e-06, 'epoch': 1.19}\n",
      "{'loss': 0.0011, 'grad_norm': 0.1248665447396375, 'learning_rate': 4.500881834215168e-06, 'epoch': 1.19}\n",
      "{'loss': 0.0, 'grad_norm': 0.00022653702500311177, 'learning_rate': 4.497354497354498e-06, 'epoch': 1.19}\n",
      "{'loss': 0.0, 'grad_norm': 0.0009692942153390522, 'learning_rate': 4.493827160493827e-06, 'epoch': 1.19}\n",
      "{'loss': 0.0, 'grad_norm': 0.00010265720013098271, 'learning_rate': 4.490299823633157e-06, 'epoch': 1.19}\n",
      "{'loss': 0.0004, 'grad_norm': 0.04237770662887455, 'learning_rate': 4.486772486772487e-06, 'epoch': 1.19}\n",
      "{'loss': 0.0, 'grad_norm': 0.0006923778982414307, 'learning_rate': 4.4832451499118165e-06, 'epoch': 1.19}\n",
      "{'loss': 0.0, 'grad_norm': 0.00027784777885629485, 'learning_rate': 4.479717813051146e-06, 'epoch': 1.2}\n",
      "{'loss': 0.0003, 'grad_norm': 0.035778787314214404, 'learning_rate': 4.476190476190477e-06, 'epoch': 1.2}\n",
      "{'loss': 0.134, 'grad_norm': 10.360449825816527, 'learning_rate': 4.4726631393298064e-06, 'epoch': 1.2}\n",
      "{'loss': 0.0, 'grad_norm': 0.0003762259153937235, 'learning_rate': 4.469135802469136e-06, 'epoch': 1.2}\n",
      "{'loss': 0.0, 'grad_norm': 6.861379434405528e-05, 'learning_rate': 4.465608465608466e-06, 'epoch': 1.2}\n",
      "{'loss': 0.3816, 'grad_norm': 20.01078429340313, 'learning_rate': 4.462081128747796e-06, 'epoch': 1.2}\n",
      "{'loss': 0.0, 'grad_norm': 0.005402799754416959, 'learning_rate': 4.458553791887126e-06, 'epoch': 1.2}\n",
      "{'loss': 0.0186, 'grad_norm': 3.0670757753347906, 'learning_rate': 4.455026455026456e-06, 'epoch': 1.2}\n",
      "{'loss': 0.0, 'grad_norm': 3.3848707251278397e-06, 'learning_rate': 4.4514991181657856e-06, 'epoch': 1.2}\n",
      "{'loss': 0.0001, 'grad_norm': 0.006652564625221815, 'learning_rate': 4.447971781305115e-06, 'epoch': 1.2}\n",
      "{'loss': 0.0, 'grad_norm': 6.854416720535083e-06, 'learning_rate': 4.444444444444444e-06, 'epoch': 1.2}\n",
      "{'loss': 0.0047, 'grad_norm': 0.6118506249237797, 'learning_rate': 4.440917107583775e-06, 'epoch': 1.2}\n",
      "{'loss': 0.0004, 'grad_norm': 0.06608658875792642, 'learning_rate': 4.437389770723104e-06, 'epoch': 1.2}\n",
      "{'loss': 0.0, 'grad_norm': 4.1763261294831867e-07, 'learning_rate': 4.433862433862434e-06, 'epoch': 1.2}\n",
      "{'loss': 0.0, 'grad_norm': 4.7791557442321277e-05, 'learning_rate': 4.430335097001764e-06, 'epoch': 1.2}\n",
      "{'loss': 0.2896, 'grad_norm': 17.3866709200926, 'learning_rate': 4.4268077601410936e-06, 'epoch': 1.21}\n",
      "{'loss': 0.0004, 'grad_norm': 0.044052727869887856, 'learning_rate': 4.423280423280424e-06, 'epoch': 1.21}\n",
      "{'loss': 0.0023, 'grad_norm': 0.25145712535086556, 'learning_rate': 4.419753086419754e-06, 'epoch': 1.21}\n",
      "{'loss': 0.0, 'grad_norm': 4.668486953269555e-05, 'learning_rate': 4.4162257495590835e-06, 'epoch': 1.21}\n",
      "{'loss': 0.0, 'grad_norm': 0.0002699572995696303, 'learning_rate': 4.412698412698413e-06, 'epoch': 1.21}\n",
      "{'loss': 0.0, 'grad_norm': 2.9188396854845996e-06, 'learning_rate': 4.409171075837743e-06, 'epoch': 1.21}\n",
      "{'loss': 0.0005, 'grad_norm': 0.06527835867917242, 'learning_rate': 4.405643738977073e-06, 'epoch': 1.21}\n",
      "{'loss': 0.0063, 'grad_norm': 0.7818197461676119, 'learning_rate': 4.402116402116402e-06, 'epoch': 1.21}\n",
      "{'loss': 0.0, 'grad_norm': 0.00024206544816647964, 'learning_rate': 4.398589065255732e-06, 'epoch': 1.21}\n",
      "{'loss': 0.0003, 'grad_norm': 0.03418126783804708, 'learning_rate': 4.395061728395062e-06, 'epoch': 1.21}\n",
      "{'loss': 0.0001, 'grad_norm': 0.004669129458521946, 'learning_rate': 4.3915343915343915e-06, 'epoch': 1.21}\n",
      "{'loss': 0.0001, 'grad_norm': 0.006700829016194982, 'learning_rate': 4.388007054673722e-06, 'epoch': 1.21}\n",
      "{'loss': 0.0, 'grad_norm': 0.002415524794237387, 'learning_rate': 4.384479717813052e-06, 'epoch': 1.21}\n",
      "{'loss': 0.0, 'grad_norm': 6.195730444453786e-05, 'learning_rate': 4.3809523809523815e-06, 'epoch': 1.21}\n",
      "{'loss': 0.0, 'grad_norm': 0.0009099670719674699, 'learning_rate': 4.377425044091711e-06, 'epoch': 1.21}\n",
      "{'loss': 0.0003, 'grad_norm': 0.03496300881659211, 'learning_rate': 4.373897707231041e-06, 'epoch': 1.21}\n",
      "{'loss': 0.0, 'grad_norm': 0.00010615611916346828, 'learning_rate': 4.370370370370371e-06, 'epoch': 1.22}\n",
      "{'loss': 0.0, 'grad_norm': 0.0032864473467893824, 'learning_rate': 4.3668430335097e-06, 'epoch': 1.22}\n",
      "{'loss': 0.0003, 'grad_norm': 0.029561732058589005, 'learning_rate': 4.36331569664903e-06, 'epoch': 1.22}\n",
      "{'loss': 0.0, 'grad_norm': 0.0011415167356451526, 'learning_rate': 4.35978835978836e-06, 'epoch': 1.22}\n",
      "{'loss': 0.0, 'grad_norm': 0.00014288646313221986, 'learning_rate': 4.3562610229276895e-06, 'epoch': 1.22}\n",
      "{'loss': 0.0, 'grad_norm': 3.245649760417686e-05, 'learning_rate': 4.35273368606702e-06, 'epoch': 1.22}\n",
      "{'loss': 0.0122, 'grad_norm': 1.937232152821435, 'learning_rate': 4.34920634920635e-06, 'epoch': 1.22}\n",
      "{'loss': 0.0, 'grad_norm': 0.0044704874907140904, 'learning_rate': 4.3456790123456795e-06, 'epoch': 1.22}\n",
      "{'loss': 0.1293, 'grad_norm': 13.307656996677174, 'learning_rate': 4.342151675485009e-06, 'epoch': 1.22}\n",
      "{'loss': 0.0007, 'grad_norm': 0.06888041494743545, 'learning_rate': 4.338624338624339e-06, 'epoch': 1.22}\n",
      "{'loss': 0.0, 'grad_norm': 0.0005412197847395355, 'learning_rate': 4.335097001763669e-06, 'epoch': 1.22}\n",
      "{'loss': 0.0266, 'grad_norm': 2.5817727841099325, 'learning_rate': 4.331569664902998e-06, 'epoch': 1.22}\n",
      "{'loss': 0.0, 'grad_norm': 0.00010880026288884076, 'learning_rate': 4.328042328042328e-06, 'epoch': 1.22}\n",
      "{'loss': 0.0002, 'grad_norm': 0.024839449210722203, 'learning_rate': 4.324514991181658e-06, 'epoch': 1.22}\n",
      "{'loss': 0.0, 'grad_norm': 1.8443272313258236e-05, 'learning_rate': 4.3209876543209875e-06, 'epoch': 1.22}\n",
      "{'loss': 0.0, 'grad_norm': 0.00011088651619429035, 'learning_rate': 4.317460317460318e-06, 'epoch': 1.22}\n",
      "{'loss': 0.0001, 'grad_norm': 0.005250269096078392, 'learning_rate': 4.313932980599648e-06, 'epoch': 1.23}\n",
      "{'loss': 0.0013, 'grad_norm': 0.151939571188003, 'learning_rate': 4.3104056437389775e-06, 'epoch': 1.23}\n",
      "{'loss': 0.0007, 'grad_norm': 0.08245356309932311, 'learning_rate': 4.306878306878307e-06, 'epoch': 1.23}\n",
      "{'loss': 0.0, 'grad_norm': 0.004637047167832956, 'learning_rate': 4.303350970017637e-06, 'epoch': 1.23}\n",
      "{'loss': 0.0, 'grad_norm': 0.000971189119649514, 'learning_rate': 4.2998236331569675e-06, 'epoch': 1.23}\n",
      "{'loss': 0.0, 'grad_norm': 0.0005122368380373602, 'learning_rate': 4.296296296296296e-06, 'epoch': 1.23}\n",
      "{'loss': 0.0, 'grad_norm': 0.00018504339488074927, 'learning_rate': 4.292768959435626e-06, 'epoch': 1.23}\n",
      "{'loss': 0.0, 'grad_norm': 0.000657366589089101, 'learning_rate': 4.289241622574956e-06, 'epoch': 1.23}\n",
      "{'loss': 0.0, 'grad_norm': 4.9786964944852884e-05, 'learning_rate': 4.2857142857142855e-06, 'epoch': 1.23}\n",
      "{'loss': 0.0, 'grad_norm': 0.00021868802832008714, 'learning_rate': 4.282186948853616e-06, 'epoch': 1.23}\n",
      "{'loss': 0.0007, 'grad_norm': 0.0904772081604893, 'learning_rate': 4.278659611992946e-06, 'epoch': 1.23}\n",
      "{'loss': 0.0, 'grad_norm': 0.0008820360021460959, 'learning_rate': 4.2751322751322754e-06, 'epoch': 1.23}\n",
      "{'loss': 0.0002, 'grad_norm': 0.01893881518624238, 'learning_rate': 4.271604938271605e-06, 'epoch': 1.23}\n",
      "{'loss': 0.0, 'grad_norm': 0.001969397730978897, 'learning_rate': 4.268077601410935e-06, 'epoch': 1.23}\n",
      "{'loss': 0.0018, 'grad_norm': 0.20145815321833052, 'learning_rate': 4.2645502645502654e-06, 'epoch': 1.23}\n",
      "{'loss': 0.002, 'grad_norm': 0.2374246082355697, 'learning_rate': 4.261022927689595e-06, 'epoch': 1.23}\n",
      "{'loss': 0.0, 'grad_norm': 0.0022546095257575506, 'learning_rate': 4.257495590828925e-06, 'epoch': 1.24}\n",
      "{'loss': 0.0, 'grad_norm': 1.390061809365733e-06, 'learning_rate': 4.2539682539682546e-06, 'epoch': 1.24}\n",
      "{'loss': 0.0001, 'grad_norm': 0.006052944366002709, 'learning_rate': 4.250440917107584e-06, 'epoch': 1.24}\n",
      "{'loss': 0.0, 'grad_norm': 2.324822919335577e-05, 'learning_rate': 4.246913580246914e-06, 'epoch': 1.24}\n",
      "{'loss': 0.0002, 'grad_norm': 0.017321005851764666, 'learning_rate': 4.243386243386244e-06, 'epoch': 1.24}\n",
      "{'loss': 0.0002, 'grad_norm': 0.024325601532022262, 'learning_rate': 4.239858906525573e-06, 'epoch': 1.24}\n",
      "{'loss': 0.0, 'grad_norm': 0.0004421215899720343, 'learning_rate': 4.236331569664903e-06, 'epoch': 1.24}\n",
      "{'loss': 0.0, 'grad_norm': 0.00026804429438893245, 'learning_rate': 4.232804232804233e-06, 'epoch': 1.24}\n",
      "{'loss': 0.0, 'grad_norm': 1.7073025352657647e-05, 'learning_rate': 4.229276895943563e-06, 'epoch': 1.24}\n",
      "{'loss': 0.0006, 'grad_norm': 0.09789036345085403, 'learning_rate': 4.225749559082893e-06, 'epoch': 1.24}\n",
      "{'loss': 0.0844, 'grad_norm': 6.891364025526747, 'learning_rate': 4.222222222222223e-06, 'epoch': 1.24}\n",
      "{'loss': 0.0, 'grad_norm': 5.147474380711694e-06, 'learning_rate': 4.2186948853615525e-06, 'epoch': 1.24}\n",
      "{'loss': 0.0, 'grad_norm': 8.684976851431176e-06, 'learning_rate': 4.215167548500882e-06, 'epoch': 1.24}\n",
      "{'loss': 0.0, 'grad_norm': 0.0012558333326420446, 'learning_rate': 4.211640211640212e-06, 'epoch': 1.24}\n",
      "{'loss': 0.0464, 'grad_norm': 3.5377585596279326, 'learning_rate': 4.208112874779542e-06, 'epoch': 1.24}\n",
      "{'loss': 0.1694, 'grad_norm': 8.834593005302278, 'learning_rate': 4.204585537918871e-06, 'epoch': 1.25}\n",
      "{'loss': 0.0, 'grad_norm': 0.00010029619486773953, 'learning_rate': 4.201058201058201e-06, 'epoch': 1.25}\n",
      "{'loss': 0.1406, 'grad_norm': 8.598729026381784, 'learning_rate': 4.197530864197531e-06, 'epoch': 1.25}\n",
      "{'loss': 0.0004, 'grad_norm': 0.04023161098559914, 'learning_rate': 4.194003527336861e-06, 'epoch': 1.25}\n",
      "{'loss': 0.0007, 'grad_norm': 0.061604709154820345, 'learning_rate': 4.190476190476191e-06, 'epoch': 1.25}\n",
      "{'loss': 0.0, 'grad_norm': 6.231791890825954e-06, 'learning_rate': 4.186948853615521e-06, 'epoch': 1.25}\n",
      "{'loss': 0.1216, 'grad_norm': 6.715676247750453, 'learning_rate': 4.1834215167548505e-06, 'epoch': 1.25}\n",
      "{'loss': 0.0, 'grad_norm': 9.513166927779785e-05, 'learning_rate': 4.17989417989418e-06, 'epoch': 1.25}\n",
      "{'loss': 0.0001, 'grad_norm': 0.007598227258660525, 'learning_rate': 4.17636684303351e-06, 'epoch': 1.25}\n",
      "{'loss': 0.0109, 'grad_norm': 1.0012566895565056, 'learning_rate': 4.17283950617284e-06, 'epoch': 1.25}\n",
      "{'loss': 0.0008, 'grad_norm': 0.08136845376635954, 'learning_rate': 4.169312169312169e-06, 'epoch': 1.25}\n",
      "{'loss': 0.0, 'grad_norm': 0.0038431840834196327, 'learning_rate': 4.165784832451499e-06, 'epoch': 1.25}\n",
      "{'loss': 0.0723, 'grad_norm': 5.015250218729842, 'learning_rate': 4.162257495590829e-06, 'epoch': 1.25}\n",
      "{'loss': 0.0002, 'grad_norm': 0.01632302585775683, 'learning_rate': 4.158730158730159e-06, 'epoch': 1.25}\n",
      "{'loss': 0.0001, 'grad_norm': 0.0068312739745384245, 'learning_rate': 4.155202821869489e-06, 'epoch': 1.25}\n",
      "{'loss': 0.0346, 'grad_norm': 2.079391437783151, 'learning_rate': 4.151675485008819e-06, 'epoch': 1.25}\n",
      "{'loss': 0.0219, 'grad_norm': 3.050332010677889, 'learning_rate': 4.1481481481481485e-06, 'epoch': 1.26}\n",
      "{'loss': 0.0, 'grad_norm': 1.857769515592765e-05, 'learning_rate': 4.144620811287478e-06, 'epoch': 1.26}\n",
      "{'loss': 0.0002, 'grad_norm': 0.023163250151843034, 'learning_rate': 4.141093474426808e-06, 'epoch': 1.26}\n",
      "{'loss': 0.0008, 'grad_norm': 0.0963764250881253, 'learning_rate': 4.137566137566138e-06, 'epoch': 1.26}\n",
      "{'loss': 0.0027, 'grad_norm': 0.281453813772577, 'learning_rate': 4.134038800705467e-06, 'epoch': 1.26}\n",
      "{'loss': 0.0, 'grad_norm': 4.102450430686009e-05, 'learning_rate': 4.130511463844797e-06, 'epoch': 1.26}\n",
      "{'loss': 0.0002, 'grad_norm': 0.026953427513470463, 'learning_rate': 4.126984126984127e-06, 'epoch': 1.26}\n",
      "{'loss': 0.0, 'grad_norm': 3.905072688147763e-05, 'learning_rate': 4.123456790123457e-06, 'epoch': 1.26}\n",
      "{'loss': 0.0001, 'grad_norm': 0.0096364040059521, 'learning_rate': 4.119929453262787e-06, 'epoch': 1.26}\n",
      "{'loss': 0.0012, 'grad_norm': 0.18850897475715941, 'learning_rate': 4.116402116402117e-06, 'epoch': 1.26}\n",
      "{'loss': 0.0, 'grad_norm': 1.7810996662126717e-05, 'learning_rate': 4.1128747795414465e-06, 'epoch': 1.26}\n",
      "{'loss': 0.0, 'grad_norm': 4.224166692858535e-06, 'learning_rate': 4.109347442680776e-06, 'epoch': 1.26}\n",
      "{'loss': 0.0046, 'grad_norm': 0.5888102742744137, 'learning_rate': 4.105820105820107e-06, 'epoch': 1.26}\n",
      "{'loss': 0.0001, 'grad_norm': 0.007243583031272278, 'learning_rate': 4.1022927689594365e-06, 'epoch': 1.26}\n",
      "{'loss': 0.0, 'grad_norm': 6.602156512133014e-05, 'learning_rate': 4.098765432098766e-06, 'epoch': 1.26}\n",
      "{'loss': 0.0, 'grad_norm': 0.0039113914750827684, 'learning_rate': 4.095238095238096e-06, 'epoch': 1.26}\n",
      "{'loss': 0.0, 'grad_norm': 0.001433085985920475, 'learning_rate': 4.091710758377425e-06, 'epoch': 1.27}\n",
      "{'loss': 0.0, 'grad_norm': 0.00011856652362894716, 'learning_rate': 4.088183421516755e-06, 'epoch': 1.27}\n",
      "{'loss': 0.0032, 'grad_norm': 0.36110348290961336, 'learning_rate': 4.084656084656085e-06, 'epoch': 1.27}\n",
      "{'loss': 0.0, 'grad_norm': 0.0006086489886284664, 'learning_rate': 4.081128747795415e-06, 'epoch': 1.27}\n",
      "{'loss': 0.0, 'grad_norm': 0.00042222043902098797, 'learning_rate': 4.0776014109347444e-06, 'epoch': 1.27}\n",
      "{'loss': 0.0, 'grad_norm': 0.001421710438335741, 'learning_rate': 4.074074074074074e-06, 'epoch': 1.27}\n",
      "{'loss': 0.0036, 'grad_norm': 0.37709892497135794, 'learning_rate': 4.070546737213405e-06, 'epoch': 1.27}\n",
      "{'loss': 0.0, 'grad_norm': 0.0003576287627208978, 'learning_rate': 4.0670194003527344e-06, 'epoch': 1.27}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 63%|██████▎   | 2000/3150 [08:21<04:30,  4.25it/s]12/23/2024 06:43:28 - INFO - FlagEmbedding.finetune.embedder.encoder_only.base.trainer -   Saving model checkpoint to ./test_encoder_only_base_bge-large-en-v1.5/checkpoint-2000\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.0001, 'grad_norm': 0.012578935167627541, 'learning_rate': 4.063492063492064e-06, 'epoch': 1.27}\n",
      "{'loss': 0.0, 'grad_norm': 4.0970670215411106e-05, 'learning_rate': 4.059964726631394e-06, 'epoch': 1.27}\n",
      "{'loss': 0.0001, 'grad_norm': 0.011658719653620064, 'learning_rate': 4.0564373897707236e-06, 'epoch': 1.27}\n",
      "{'loss': 0.0, 'grad_norm': 0.0008131945372193306, 'learning_rate': 4.052910052910053e-06, 'epoch': 1.27}\n",
      "{'loss': 0.0, 'grad_norm': 0.00026534978358113776, 'learning_rate': 4.049382716049383e-06, 'epoch': 1.27}\n",
      "{'loss': 0.0, 'grad_norm': 1.1366631129487001e-05, 'learning_rate': 4.045855379188713e-06, 'epoch': 1.27}\n",
      "{'loss': 0.0116, 'grad_norm': 1.3234954028653214, 'learning_rate': 4.042328042328042e-06, 'epoch': 1.27}\n",
      "{'loss': 0.0004, 'grad_norm': 0.05145979726251188, 'learning_rate': 4.038800705467372e-06, 'epoch': 1.27}\n",
      "{'loss': 0.0, 'grad_norm': 0.0003372150780671462, 'learning_rate': 4.035273368606703e-06, 'epoch': 1.28}\n",
      "{'loss': 0.0, 'grad_norm': 0.0016526051184216895, 'learning_rate': 4.031746031746032e-06, 'epoch': 1.28}\n",
      "{'loss': 0.0, 'grad_norm': 0.00046429687595757763, 'learning_rate': 4.028218694885362e-06, 'epoch': 1.28}\n",
      "{'loss': 0.0001, 'grad_norm': 0.009080742416403165, 'learning_rate': 4.024691358024692e-06, 'epoch': 1.28}\n",
      "{'loss': 0.0, 'grad_norm': 4.170996410800277e-07, 'learning_rate': 4.0211640211640215e-06, 'epoch': 1.28}\n",
      "{'loss': 0.0001, 'grad_norm': 0.0056341060816663254, 'learning_rate': 4.017636684303351e-06, 'epoch': 1.28}\n",
      "{'loss': 0.0, 'grad_norm': 0.0003476837383823824, 'learning_rate': 4.014109347442681e-06, 'epoch': 1.28}\n",
      "{'loss': 0.0, 'grad_norm': 1.2932456203051463e-05, 'learning_rate': 4.010582010582011e-06, 'epoch': 1.28}\n",
      "{'loss': 0.0, 'grad_norm': 0.001667054280237365, 'learning_rate': 4.00705467372134e-06, 'epoch': 1.28}\n",
      "{'loss': 0.0305, 'grad_norm': 3.214005909200386, 'learning_rate': 4.00352733686067e-06, 'epoch': 1.28}\n",
      "{'loss': 0.1406, 'grad_norm': 9.957375665262417, 'learning_rate': 4.000000000000001e-06, 'epoch': 1.28}\n",
      "{'loss': 0.0044, 'grad_norm': 0.4382931807962962, 'learning_rate': 3.99647266313933e-06, 'epoch': 1.28}\n",
      "{'loss': 0.0005, 'grad_norm': 0.044078655069264686, 'learning_rate': 3.99294532627866e-06, 'epoch': 1.28}\n",
      "{'loss': 0.0, 'grad_norm': 1.76998786380777e-06, 'learning_rate': 3.98941798941799e-06, 'epoch': 1.28}\n",
      "{'loss': 0.0001, 'grad_norm': 0.00978272029518602, 'learning_rate': 3.9858906525573195e-06, 'epoch': 1.28}\n",
      "{'loss': 0.0009, 'grad_norm': 0.1130127025644626, 'learning_rate': 3.982363315696649e-06, 'epoch': 1.29}\n",
      "{'loss': 0.0, 'grad_norm': 0.00027384888143693466, 'learning_rate': 3.978835978835979e-06, 'epoch': 1.29}\n",
      "{'loss': 0.0, 'grad_norm': 0.0001357501575099601, 'learning_rate': 3.975308641975309e-06, 'epoch': 1.29}\n",
      "{'loss': 0.0, 'grad_norm': 0.0018561665323238428, 'learning_rate': 3.971781305114638e-06, 'epoch': 1.29}\n",
      "{'loss': 0.0, 'grad_norm': 0.00011872288444423105, 'learning_rate': 3.968253968253968e-06, 'epoch': 1.29}\n",
      "{'loss': 0.0292, 'grad_norm': 2.48721189891218, 'learning_rate': 3.964726631393299e-06, 'epoch': 1.29}\n",
      "{'loss': 0.0104, 'grad_norm': 1.222874777280617, 'learning_rate': 3.961199294532628e-06, 'epoch': 1.29}\n",
      "{'loss': 0.0001, 'grad_norm': 0.009064794325900323, 'learning_rate': 3.957671957671958e-06, 'epoch': 1.29}\n",
      "{'loss': 0.0001, 'grad_norm': 0.009378155674530883, 'learning_rate': 3.954144620811288e-06, 'epoch': 1.29}\n",
      "{'loss': 0.0296, 'grad_norm': 3.8792426970857425, 'learning_rate': 3.9506172839506175e-06, 'epoch': 1.29}\n",
      "{'loss': 0.0, 'grad_norm': 5.383315492547169e-06, 'learning_rate': 3.947089947089948e-06, 'epoch': 1.29}\n",
      "{'loss': 0.0, 'grad_norm': 0.00321624453984528, 'learning_rate': 3.943562610229277e-06, 'epoch': 1.29}\n",
      "{'loss': 0.0, 'grad_norm': 0.000508901259737745, 'learning_rate': 3.940035273368607e-06, 'epoch': 1.29}\n",
      "{'loss': 0.0, 'grad_norm': 6.175520505608085e-05, 'learning_rate': 3.936507936507936e-06, 'epoch': 1.29}\n",
      "{'loss': 0.0, 'grad_norm': 3.328684466498511e-06, 'learning_rate': 3.932980599647266e-06, 'epoch': 1.29}\n",
      "{'loss': 0.0076, 'grad_norm': 0.496943504481454, 'learning_rate': 3.929453262786597e-06, 'epoch': 1.29}\n",
      "{'loss': 0.0347, 'grad_norm': 3.5131371405567666, 'learning_rate': 3.925925925925926e-06, 'epoch': 1.3}\n",
      "{'loss': 0.0, 'grad_norm': 0.002503422782381375, 'learning_rate': 3.922398589065256e-06, 'epoch': 1.3}\n",
      "{'loss': 0.0001, 'grad_norm': 0.011731753361970684, 'learning_rate': 3.918871252204586e-06, 'epoch': 1.3}\n",
      "{'loss': 0.0, 'grad_norm': 0.0038656862268593075, 'learning_rate': 3.9153439153439155e-06, 'epoch': 1.3}\n",
      "{'loss': 0.0, 'grad_norm': 0.0002318315973215827, 'learning_rate': 3.911816578483246e-06, 'epoch': 1.3}\n",
      "{'loss': 0.0, 'grad_norm': 0.002216825025707688, 'learning_rate': 3.908289241622576e-06, 'epoch': 1.3}\n",
      "{'loss': 0.0, 'grad_norm': 0.00029305770302495547, 'learning_rate': 3.9047619047619055e-06, 'epoch': 1.3}\n",
      "{'loss': 0.0084, 'grad_norm': 0.963058642726068, 'learning_rate': 3.901234567901235e-06, 'epoch': 1.3}\n",
      "{'loss': 0.0015, 'grad_norm': 0.18851907280347097, 'learning_rate': 3.897707231040565e-06, 'epoch': 1.3}\n",
      "{'loss': 0.0, 'grad_norm': 0.0001246165642187544, 'learning_rate': 3.894179894179895e-06, 'epoch': 1.3}\n",
      "{'loss': 0.0002, 'grad_norm': 0.02998428052565057, 'learning_rate': 3.890652557319224e-06, 'epoch': 1.3}\n",
      "{'loss': 0.0003, 'grad_norm': 0.04066659518452237, 'learning_rate': 3.887125220458554e-06, 'epoch': 1.3}\n",
      "{'loss': 0.0, 'grad_norm': 0.0023534603119816236, 'learning_rate': 3.883597883597884e-06, 'epoch': 1.3}\n",
      "{'loss': 0.0, 'grad_norm': 0.001112300225116578, 'learning_rate': 3.8800705467372134e-06, 'epoch': 1.3}\n",
      "{'loss': 0.0, 'grad_norm': 7.118388108184567e-05, 'learning_rate': 3.876543209876544e-06, 'epoch': 1.3}\n",
      "{'loss': 0.0, 'grad_norm': 4.165418900310133e-06, 'learning_rate': 3.873015873015874e-06, 'epoch': 1.3}\n",
      "{'loss': 0.0, 'grad_norm': 7.288626385252511e-06, 'learning_rate': 3.8694885361552034e-06, 'epoch': 1.31}\n",
      "{'loss': 0.0, 'grad_norm': 0.00029729638874398845, 'learning_rate': 3.865961199294533e-06, 'epoch': 1.31}\n",
      "{'loss': 0.0003, 'grad_norm': 0.05246448550649802, 'learning_rate': 3.862433862433863e-06, 'epoch': 1.31}\n",
      "{'loss': 0.0, 'grad_norm': 0.0013518195092706055, 'learning_rate': 3.8589065255731926e-06, 'epoch': 1.31}\n",
      "{'loss': 0.0032, 'grad_norm': 0.3448976843052749, 'learning_rate': 3.855379188712522e-06, 'epoch': 1.31}\n",
      "{'loss': 0.0, 'grad_norm': 1.0908658880027973e-05, 'learning_rate': 3.851851851851852e-06, 'epoch': 1.31}\n",
      "{'loss': 0.0, 'grad_norm': 0.0010983940575369172, 'learning_rate': 3.848324514991182e-06, 'epoch': 1.31}\n",
      "{'loss': 0.0872, 'grad_norm': 11.253861506332434, 'learning_rate': 3.844797178130511e-06, 'epoch': 1.31}\n",
      "{'loss': 0.0, 'grad_norm': 7.999734566921544e-06, 'learning_rate': 3.841269841269842e-06, 'epoch': 1.31}\n",
      "{'loss': 0.1824, 'grad_norm': 8.19363895343921, 'learning_rate': 3.837742504409172e-06, 'epoch': 1.31}\n",
      "{'loss': 0.0, 'grad_norm': 0.0006462749540247018, 'learning_rate': 3.834215167548501e-06, 'epoch': 1.31}\n",
      "{'loss': 0.0, 'grad_norm': 8.756691179607897e-05, 'learning_rate': 3.830687830687831e-06, 'epoch': 1.31}\n",
      "{'loss': 0.0125, 'grad_norm': 1.2095130664171687, 'learning_rate': 3.827160493827161e-06, 'epoch': 1.31}\n",
      "{'loss': 0.001, 'grad_norm': 0.14274687774386957, 'learning_rate': 3.8236331569664905e-06, 'epoch': 1.31}\n",
      "{'loss': 0.0, 'grad_norm': 0.000388774154686567, 'learning_rate': 3.82010582010582e-06, 'epoch': 1.31}\n",
      "{'loss': 0.0002, 'grad_norm': 0.026227646934749632, 'learning_rate': 3.81657848324515e-06, 'epoch': 1.31}\n",
      "{'loss': 0.0011, 'grad_norm': 0.11808142254974845, 'learning_rate': 3.81305114638448e-06, 'epoch': 1.32}\n",
      "{'loss': 0.0002, 'grad_norm': 0.01919913448109571, 'learning_rate': 3.80952380952381e-06, 'epoch': 1.32}\n",
      "{'loss': 0.0, 'grad_norm': 0.0004083531918304264, 'learning_rate': 3.80599647266314e-06, 'epoch': 1.32}\n",
      "{'loss': 0.0, 'grad_norm': 0.0015984192514464154, 'learning_rate': 3.8024691358024697e-06, 'epoch': 1.32}\n",
      "{'loss': 0.0, 'grad_norm': 0.0005278615878233923, 'learning_rate': 3.7989417989417994e-06, 'epoch': 1.32}\n",
      "{'loss': 0.0, 'grad_norm': 0.0005033259586935129, 'learning_rate': 3.795414462081129e-06, 'epoch': 1.32}\n",
      "{'loss': 0.0, 'grad_norm': 0.00039891621408594323, 'learning_rate': 3.791887125220459e-06, 'epoch': 1.32}\n",
      "{'loss': 0.0, 'grad_norm': 6.448488805168442e-06, 'learning_rate': 3.788359788359789e-06, 'epoch': 1.32}\n",
      "{'loss': 0.0, 'grad_norm': 0.0007266480056301328, 'learning_rate': 3.7848324514991187e-06, 'epoch': 1.32}\n",
      "{'loss': 0.0, 'grad_norm': 0.000841282650921329, 'learning_rate': 3.7813051146384484e-06, 'epoch': 1.32}\n",
      "{'loss': 0.0251, 'grad_norm': 2.2022602738023758, 'learning_rate': 3.777777777777778e-06, 'epoch': 1.32}\n",
      "{'loss': 0.0, 'grad_norm': 0.004260186034976314, 'learning_rate': 3.774250440917108e-06, 'epoch': 1.32}\n",
      "{'loss': 0.0, 'grad_norm': 3.432160879665568e-05, 'learning_rate': 3.770723104056438e-06, 'epoch': 1.32}\n",
      "{'loss': 0.0, 'grad_norm': 4.821083028809395e-07, 'learning_rate': 3.7671957671957676e-06, 'epoch': 1.32}\n",
      "{'loss': 0.0, 'grad_norm': 6.36190448945569e-05, 'learning_rate': 3.7636684303350974e-06, 'epoch': 1.32}\n",
      "{'loss': 0.0, 'grad_norm': 0.0007083436448469876, 'learning_rate': 3.760141093474427e-06, 'epoch': 1.33}\n",
      "{'loss': 0.1307, 'grad_norm': 8.995499651328478, 'learning_rate': 3.7566137566137568e-06, 'epoch': 1.33}\n",
      "{'loss': 0.0, 'grad_norm': 0.00011540753771249655, 'learning_rate': 3.753086419753087e-06, 'epoch': 1.33}\n",
      "{'loss': 0.0, 'grad_norm': 0.00017934188786238698, 'learning_rate': 3.7495590828924166e-06, 'epoch': 1.33}\n",
      "{'loss': 0.0, 'grad_norm': 0.0026630256018836925, 'learning_rate': 3.7460317460317463e-06, 'epoch': 1.33}\n",
      "{'loss': 0.0, 'grad_norm': 0.0019788383572559905, 'learning_rate': 3.742504409171076e-06, 'epoch': 1.33}\n",
      "{'loss': 0.0, 'grad_norm': 7.269220805473827e-05, 'learning_rate': 3.7389770723104058e-06, 'epoch': 1.33}\n",
      "{'loss': 0.0, 'grad_norm': 0.0001803610254544812, 'learning_rate': 3.735449735449736e-06, 'epoch': 1.33}\n",
      "{'loss': 1.0273, 'grad_norm': 34.462077209956455, 'learning_rate': 3.7319223985890656e-06, 'epoch': 1.33}\n",
      "{'loss': 0.0001, 'grad_norm': 0.006422686905680291, 'learning_rate': 3.7283950617283953e-06, 'epoch': 1.33}\n",
      "{'loss': 0.0002, 'grad_norm': 0.017163179778524877, 'learning_rate': 3.724867724867725e-06, 'epoch': 1.33}\n",
      "{'loss': 0.0006, 'grad_norm': 0.09415983977439693, 'learning_rate': 3.7213403880070548e-06, 'epoch': 1.33}\n",
      "{'loss': 0.0688, 'grad_norm': 8.731934563049526, 'learning_rate': 3.717813051146385e-06, 'epoch': 1.33}\n",
      "{'loss': 0.0135, 'grad_norm': 1.595661064723562, 'learning_rate': 3.7142857142857146e-06, 'epoch': 1.33}\n",
      "{'loss': 0.0, 'grad_norm': 0.002380051843618939, 'learning_rate': 3.7107583774250443e-06, 'epoch': 1.33}\n",
      "{'loss': 0.0, 'grad_norm': 8.795381204501326e-06, 'learning_rate': 3.707231040564374e-06, 'epoch': 1.33}\n",
      "{'loss': 0.0, 'grad_norm': 0.0007258834831123965, 'learning_rate': 3.7037037037037037e-06, 'epoch': 1.34}\n",
      "{'loss': 0.0, 'grad_norm': 0.003228323495501856, 'learning_rate': 3.700176366843034e-06, 'epoch': 1.34}\n",
      "{'loss': 0.0002, 'grad_norm': 0.031197199695483818, 'learning_rate': 3.6966490299823636e-06, 'epoch': 1.34}\n",
      "{'loss': 0.0, 'grad_norm': 0.001380782793956965, 'learning_rate': 3.6931216931216933e-06, 'epoch': 1.34}\n",
      "{'loss': 0.0, 'grad_norm': 0.001530721995351247, 'learning_rate': 3.689594356261023e-06, 'epoch': 1.34}\n",
      "{'loss': 0.0004, 'grad_norm': 0.04428473364231408, 'learning_rate': 3.6860670194003527e-06, 'epoch': 1.34}\n",
      "{'loss': 0.0, 'grad_norm': 0.0023925967627828498, 'learning_rate': 3.6825396825396833e-06, 'epoch': 1.34}\n",
      "{'loss': 0.9658, 'grad_norm': 31.80079828375972, 'learning_rate': 3.6790123456790126e-06, 'epoch': 1.34}\n",
      "{'loss': 0.0, 'grad_norm': 0.0031061334993138182, 'learning_rate': 3.6754850088183423e-06, 'epoch': 1.34}\n",
      "{'loss': 0.0013, 'grad_norm': 0.1089067226903172, 'learning_rate': 3.671957671957672e-06, 'epoch': 1.34}\n",
      "{'loss': 0.0003, 'grad_norm': 0.02421263451235768, 'learning_rate': 3.6684303350970017e-06, 'epoch': 1.34}\n",
      "{'loss': 0.0, 'grad_norm': 0.000683460967924195, 'learning_rate': 3.6649029982363323e-06, 'epoch': 1.34}\n",
      "{'loss': 0.0013, 'grad_norm': 0.21596093580592832, 'learning_rate': 3.661375661375662e-06, 'epoch': 1.34}\n",
      "{'loss': 0.0, 'grad_norm': 4.370775519172615e-05, 'learning_rate': 3.6578483245149917e-06, 'epoch': 1.34}\n",
      "{'loss': 0.0, 'grad_norm': 1.8889832940890255e-06, 'learning_rate': 3.654320987654321e-06, 'epoch': 1.34}\n",
      "{'loss': 0.0001, 'grad_norm': 0.014771209258771224, 'learning_rate': 3.6507936507936507e-06, 'epoch': 1.34}\n",
      "{'loss': 0.0, 'grad_norm': 0.00019942313045017841, 'learning_rate': 3.6472663139329813e-06, 'epoch': 1.35}\n",
      "{'loss': 0.0, 'grad_norm': 9.002777475047074e-05, 'learning_rate': 3.643738977072311e-06, 'epoch': 1.35}\n",
      "{'loss': 0.0, 'grad_norm': 0.0027423125932599557, 'learning_rate': 3.6402116402116407e-06, 'epoch': 1.35}\n",
      "{'loss': 0.0, 'grad_norm': 0.00014894959232002902, 'learning_rate': 3.6366843033509704e-06, 'epoch': 1.35}\n",
      "{'loss': 0.0, 'grad_norm': 9.87332229742095e-05, 'learning_rate': 3.6331569664903e-06, 'epoch': 1.35}\n",
      "{'loss': 0.0, 'grad_norm': 0.0012501254881811031, 'learning_rate': 3.6296296296296302e-06, 'epoch': 1.35}\n",
      "{'loss': 0.0, 'grad_norm': 0.00018329022366889372, 'learning_rate': 3.62610229276896e-06, 'epoch': 1.35}\n",
      "{'loss': 0.0328, 'grad_norm': 4.230683353859634, 'learning_rate': 3.6225749559082897e-06, 'epoch': 1.35}\n",
      "{'loss': 0.0031, 'grad_norm': 0.34273090805133205, 'learning_rate': 3.6190476190476194e-06, 'epoch': 1.35}\n",
      "{'loss': 0.0002, 'grad_norm': 0.021415685987305675, 'learning_rate': 3.615520282186949e-06, 'epoch': 1.35}\n",
      "{'loss': 0.0, 'grad_norm': 0.0019389557427290525, 'learning_rate': 3.6119929453262792e-06, 'epoch': 1.35}\n",
      "{'loss': 0.0002, 'grad_norm': 0.02012603186887081, 'learning_rate': 3.608465608465609e-06, 'epoch': 1.35}\n",
      "{'loss': 0.0034, 'grad_norm': 0.5161642953423848, 'learning_rate': 3.6049382716049387e-06, 'epoch': 1.35}\n",
      "{'loss': 0.0001, 'grad_norm': 0.0066537019834517815, 'learning_rate': 3.6014109347442684e-06, 'epoch': 1.35}\n",
      "{'loss': 0.0, 'grad_norm': 0.0033826544346143026, 'learning_rate': 3.597883597883598e-06, 'epoch': 1.35}\n",
      "{'loss': 0.2269, 'grad_norm': 19.92534053803449, 'learning_rate': 3.5943562610229282e-06, 'epoch': 1.35}\n",
      "{'loss': 0.0166, 'grad_norm': 2.3078458190868196, 'learning_rate': 3.590828924162258e-06, 'epoch': 1.36}\n",
      "{'loss': 0.0, 'grad_norm': 7.892198285540463e-06, 'learning_rate': 3.5873015873015877e-06, 'epoch': 1.36}\n",
      "{'loss': 0.0008, 'grad_norm': 0.09287889409725965, 'learning_rate': 3.5837742504409174e-06, 'epoch': 1.36}\n",
      "{'loss': 0.0, 'grad_norm': 4.0712020421593737e-05, 'learning_rate': 3.580246913580247e-06, 'epoch': 1.36}\n",
      "{'loss': 0.0, 'grad_norm': 8.583946364130694e-06, 'learning_rate': 3.5767195767195772e-06, 'epoch': 1.36}\n",
      "{'loss': 0.0316, 'grad_norm': 4.551620052457006, 'learning_rate': 3.573192239858907e-06, 'epoch': 1.36}\n",
      "{'loss': 0.0, 'grad_norm': 0.0003815565345758032, 'learning_rate': 3.5696649029982366e-06, 'epoch': 1.36}\n",
      "{'loss': 0.0044, 'grad_norm': 0.5463261711383511, 'learning_rate': 3.5661375661375664e-06, 'epoch': 1.36}\n",
      "{'loss': 0.0001, 'grad_norm': 0.008552470449292058, 'learning_rate': 3.562610229276896e-06, 'epoch': 1.36}\n",
      "{'loss': 0.0, 'grad_norm': 0.004212763649829117, 'learning_rate': 3.559082892416226e-06, 'epoch': 1.36}\n",
      "{'loss': 0.0, 'grad_norm': 0.00028946154857470744, 'learning_rate': 3.555555555555556e-06, 'epoch': 1.36}\n",
      "{'loss': 0.0206, 'grad_norm': 2.7357981438822976, 'learning_rate': 3.5520282186948856e-06, 'epoch': 1.36}\n",
      "{'loss': 0.0, 'grad_norm': 0.0001278162786396002, 'learning_rate': 3.5485008818342153e-06, 'epoch': 1.36}\n",
      "{'loss': 0.0005, 'grad_norm': 0.04972838887264093, 'learning_rate': 3.544973544973545e-06, 'epoch': 1.36}\n",
      "{'loss': 0.0002, 'grad_norm': 0.015296316728874127, 'learning_rate': 3.541446208112875e-06, 'epoch': 1.36}\n",
      "{'loss': 0.0009, 'grad_norm': 0.14092512687537312, 'learning_rate': 3.537918871252205e-06, 'epoch': 1.37}\n",
      "{'loss': 0.0002, 'grad_norm': 0.02885679479908465, 'learning_rate': 3.5343915343915346e-06, 'epoch': 1.37}\n",
      "{'loss': 0.0034, 'grad_norm': 0.38602396656651794, 'learning_rate': 3.5308641975308643e-06, 'epoch': 1.37}\n",
      "{'loss': 0.0001, 'grad_norm': 0.019426914276060962, 'learning_rate': 3.527336860670194e-06, 'epoch': 1.37}\n",
      "{'loss': 0.0, 'grad_norm': 0.004924574344874393, 'learning_rate': 3.523809523809524e-06, 'epoch': 1.37}\n",
      "{'loss': 0.0, 'grad_norm': 0.0022079770449268767, 'learning_rate': 3.520282186948854e-06, 'epoch': 1.37}\n",
      "{'loss': 0.0, 'grad_norm': 7.454383147965024e-06, 'learning_rate': 3.5167548500881836e-06, 'epoch': 1.37}\n",
      "{'loss': 0.0009, 'grad_norm': 0.09799336420797465, 'learning_rate': 3.5132275132275133e-06, 'epoch': 1.37}\n",
      "{'loss': 0.0002, 'grad_norm': 0.01894259977213991, 'learning_rate': 3.509700176366843e-06, 'epoch': 1.37}\n",
      "{'loss': 0.0, 'grad_norm': 0.0002455709628687865, 'learning_rate': 3.5061728395061736e-06, 'epoch': 1.37}\n",
      "{'loss': 0.0, 'grad_norm': 3.869823529062905e-05, 'learning_rate': 3.502645502645503e-06, 'epoch': 1.37}\n",
      "{'loss': 0.0848, 'grad_norm': 3.6774558652178273, 'learning_rate': 3.4991181657848326e-06, 'epoch': 1.37}\n",
      "{'loss': 0.0016, 'grad_norm': 0.20679871978052217, 'learning_rate': 3.4955908289241623e-06, 'epoch': 1.37}\n",
      "{'loss': 0.0, 'grad_norm': 2.523118960808546e-05, 'learning_rate': 3.492063492063492e-06, 'epoch': 1.37}\n",
      "{'loss': 0.0001, 'grad_norm': 0.014769830964083502, 'learning_rate': 3.4885361552028226e-06, 'epoch': 1.37}\n",
      "{'loss': 0.0014, 'grad_norm': 0.16784974806321798, 'learning_rate': 3.4850088183421523e-06, 'epoch': 1.37}\n",
      "{'loss': 0.0001, 'grad_norm': 0.007322383938882444, 'learning_rate': 3.481481481481482e-06, 'epoch': 1.38}\n",
      "{'loss': 0.0, 'grad_norm': 0.00011297605629770727, 'learning_rate': 3.4779541446208113e-06, 'epoch': 1.38}\n",
      "{'loss': 0.0001, 'grad_norm': 0.00773364345347797, 'learning_rate': 3.474426807760141e-06, 'epoch': 1.38}\n",
      "{'loss': 0.0, 'grad_norm': 1.1868507709978623e-05, 'learning_rate': 3.4708994708994716e-06, 'epoch': 1.38}\n",
      "{'loss': 0.0064, 'grad_norm': 0.885981551593972, 'learning_rate': 3.4673721340388013e-06, 'epoch': 1.38}\n",
      "{'loss': 0.0, 'grad_norm': 0.0018606387547388322, 'learning_rate': 3.463844797178131e-06, 'epoch': 1.38}\n",
      "{'loss': 0.0036, 'grad_norm': 0.5866502622817878, 'learning_rate': 3.4603174603174607e-06, 'epoch': 1.38}\n",
      "{'loss': 0.0, 'grad_norm': 0.00011129705525857995, 'learning_rate': 3.4567901234567904e-06, 'epoch': 1.38}\n",
      "{'loss': 0.0, 'grad_norm': 0.00048054690021675193, 'learning_rate': 3.4532627865961205e-06, 'epoch': 1.38}\n",
      "{'loss': 0.0, 'grad_norm': 5.4479906773283594e-05, 'learning_rate': 3.4497354497354503e-06, 'epoch': 1.38}\n",
      "{'loss': 0.0, 'grad_norm': 6.524530220331813e-05, 'learning_rate': 3.44620811287478e-06, 'epoch': 1.38}\n",
      "{'loss': 0.0, 'grad_norm': 7.574623474907023e-06, 'learning_rate': 3.4426807760141097e-06, 'epoch': 1.38}\n",
      "{'loss': 0.0, 'grad_norm': 0.00015617055620535135, 'learning_rate': 3.4391534391534394e-06, 'epoch': 1.38}\n",
      "{'loss': 0.0, 'grad_norm': 0.0007285727326130159, 'learning_rate': 3.4356261022927695e-06, 'epoch': 1.38}\n",
      "{'loss': 0.0, 'grad_norm': 7.890164148593616e-05, 'learning_rate': 3.4320987654320992e-06, 'epoch': 1.38}\n",
      "{'loss': 0.0, 'grad_norm': 0.0001415590235919119, 'learning_rate': 3.428571428571429e-06, 'epoch': 1.38}\n",
      "{'loss': 0.0, 'grad_norm': 3.2100789970894784e-05, 'learning_rate': 3.4250440917107587e-06, 'epoch': 1.39}\n",
      "{'loss': 0.0063, 'grad_norm': 0.8066291346435892, 'learning_rate': 3.4215167548500884e-06, 'epoch': 1.39}\n",
      "{'loss': 0.0007, 'grad_norm': 0.11675392049949962, 'learning_rate': 3.4179894179894185e-06, 'epoch': 1.39}\n",
      "{'loss': 0.0, 'grad_norm': 8.465136554248184e-05, 'learning_rate': 3.4144620811287482e-06, 'epoch': 1.39}\n",
      "{'loss': 0.0015, 'grad_norm': 0.20231620534815764, 'learning_rate': 3.410934744268078e-06, 'epoch': 1.39}\n",
      "{'loss': 0.0, 'grad_norm': 7.434836251704093e-06, 'learning_rate': 3.4074074074074077e-06, 'epoch': 1.39}\n",
      "{'loss': 0.0, 'grad_norm': 0.004006735044829069, 'learning_rate': 3.4038800705467374e-06, 'epoch': 1.39}\n",
      "{'loss': 0.0, 'grad_norm': 3.042435532373723e-05, 'learning_rate': 3.4003527336860675e-06, 'epoch': 1.39}\n",
      "{'loss': 0.0009, 'grad_norm': 0.08735682191770028, 'learning_rate': 3.3968253968253972e-06, 'epoch': 1.39}\n",
      "{'loss': 0.0, 'grad_norm': 0.00017244070299005085, 'learning_rate': 3.393298059964727e-06, 'epoch': 1.39}\n",
      "{'loss': 0.0, 'grad_norm': 0.003463269166343604, 'learning_rate': 3.3897707231040566e-06, 'epoch': 1.39}\n",
      "{'loss': 0.0, 'grad_norm': 0.0017074818237754856, 'learning_rate': 3.3862433862433864e-06, 'epoch': 1.39}\n",
      "{'loss': 0.0, 'grad_norm': 7.253805069374262e-06, 'learning_rate': 3.3827160493827165e-06, 'epoch': 1.39}\n",
      "{'loss': 0.0, 'grad_norm': 0.0005140038051667341, 'learning_rate': 3.379188712522046e-06, 'epoch': 1.39}\n",
      "{'loss': 0.0001, 'grad_norm': 0.006698987467821694, 'learning_rate': 3.375661375661376e-06, 'epoch': 1.39}\n",
      "{'loss': 0.0, 'grad_norm': 0.0013248304536722798, 'learning_rate': 3.3721340388007056e-06, 'epoch': 1.39}\n",
      "{'loss': 0.0002, 'grad_norm': 0.021626747824506527, 'learning_rate': 3.3686067019400353e-06, 'epoch': 1.4}\n",
      "{'loss': 0.0, 'grad_norm': 0.0002584826738414308, 'learning_rate': 3.3650793650793655e-06, 'epoch': 1.4}\n",
      "{'loss': 0.0, 'grad_norm': 0.0009365238991523105, 'learning_rate': 3.361552028218695e-06, 'epoch': 1.4}\n",
      "{'loss': 0.0, 'grad_norm': 0.0002118054972896326, 'learning_rate': 3.358024691358025e-06, 'epoch': 1.4}\n",
      "{'loss': 0.0, 'grad_norm': 1.5237730927500084e-06, 'learning_rate': 3.3544973544973546e-06, 'epoch': 1.4}\n",
      "{'loss': 0.0, 'grad_norm': 0.0003392474060153105, 'learning_rate': 3.3509700176366843e-06, 'epoch': 1.4}\n",
      "{'loss': 0.0004, 'grad_norm': 0.05362289755902389, 'learning_rate': 3.3474426807760145e-06, 'epoch': 1.4}\n",
      "{'loss': 0.0, 'grad_norm': 0.00034013556486311715, 'learning_rate': 3.343915343915344e-06, 'epoch': 1.4}\n",
      "{'loss': 0.3657, 'grad_norm': 22.85611866972455, 'learning_rate': 3.340388007054674e-06, 'epoch': 1.4}\n",
      "{'loss': 0.0, 'grad_norm': 0.0023699754661469404, 'learning_rate': 3.3368606701940036e-06, 'epoch': 1.4}\n",
      "{'loss': 0.007, 'grad_norm': 0.8282786532672389, 'learning_rate': 3.3333333333333333e-06, 'epoch': 1.4}\n",
      "{'loss': 0.0, 'grad_norm': 6.134207029008878e-05, 'learning_rate': 3.3298059964726635e-06, 'epoch': 1.4}\n",
      "{'loss': 0.0007, 'grad_norm': 0.07422478550914971, 'learning_rate': 3.326278659611993e-06, 'epoch': 1.4}\n",
      "{'loss': 0.0, 'grad_norm': 5.901144396688934e-06, 'learning_rate': 3.322751322751323e-06, 'epoch': 1.4}\n",
      "{'loss': 0.0, 'grad_norm': 0.00025462705206333834, 'learning_rate': 3.3192239858906526e-06, 'epoch': 1.4}\n",
      "{'loss': 0.0, 'grad_norm': 3.47899396553051e-06, 'learning_rate': 3.3156966490299823e-06, 'epoch': 1.41}\n",
      "{'loss': 0.0, 'grad_norm': 0.00042801892469743354, 'learning_rate': 3.312169312169313e-06, 'epoch': 1.41}\n",
      "{'loss': 0.0015, 'grad_norm': 0.23767491882403485, 'learning_rate': 3.3086419753086426e-06, 'epoch': 1.41}\n",
      "{'loss': 0.0, 'grad_norm': 4.1568292644456344e-07, 'learning_rate': 3.3051146384479723e-06, 'epoch': 1.41}\n",
      "{'loss': 0.0656, 'grad_norm': 5.718211998152093, 'learning_rate': 3.3015873015873016e-06, 'epoch': 1.41}\n",
      "{'loss': 0.0, 'grad_norm': 6.579545249986309e-05, 'learning_rate': 3.2980599647266313e-06, 'epoch': 1.41}\n",
      "{'loss': 0.0001, 'grad_norm': 0.006902647795653524, 'learning_rate': 3.294532627865962e-06, 'epoch': 1.41}\n",
      "{'loss': 0.0, 'grad_norm': 0.005573621591435287, 'learning_rate': 3.2910052910052916e-06, 'epoch': 1.41}\n",
      "{'loss': 0.0003, 'grad_norm': 0.04215338865486932, 'learning_rate': 3.2874779541446213e-06, 'epoch': 1.41}\n",
      "{'loss': 0.0, 'grad_norm': 2.8652864909777126e-05, 'learning_rate': 3.283950617283951e-06, 'epoch': 1.41}\n",
      "{'loss': 0.0001, 'grad_norm': 0.006844466021701656, 'learning_rate': 3.2804232804232807e-06, 'epoch': 1.41}\n",
      "{'loss': 0.0003, 'grad_norm': 0.02097578369369572, 'learning_rate': 3.276895943562611e-06, 'epoch': 1.41}\n",
      "{'loss': 0.0002, 'grad_norm': 0.027907623357274973, 'learning_rate': 3.2733686067019406e-06, 'epoch': 1.41}\n",
      "{'loss': 0.0, 'grad_norm': 0.0002677902234329869, 'learning_rate': 3.2698412698412703e-06, 'epoch': 1.41}\n",
      "{'loss': 0.0003, 'grad_norm': 0.046578857625659296, 'learning_rate': 3.2663139329806e-06, 'epoch': 1.41}\n",
      "{'loss': 0.0175, 'grad_norm': 1.7768318529019391, 'learning_rate': 3.2627865961199297e-06, 'epoch': 1.41}\n",
      "{'loss': 0.0001, 'grad_norm': 0.01596309651067365, 'learning_rate': 3.25925925925926e-06, 'epoch': 1.42}\n",
      "{'loss': 0.0088, 'grad_norm': 1.0414267581600227, 'learning_rate': 3.2557319223985895e-06, 'epoch': 1.42}\n",
      "{'loss': 0.0, 'grad_norm': 0.00045817342563871067, 'learning_rate': 3.2522045855379193e-06, 'epoch': 1.42}\n",
      "{'loss': 0.0, 'grad_norm': 0.0008257969860453905, 'learning_rate': 3.248677248677249e-06, 'epoch': 1.42}\n",
      "{'loss': 0.0, 'grad_norm': 0.00030828709277101003, 'learning_rate': 3.2451499118165787e-06, 'epoch': 1.42}\n",
      "{'loss': 0.0, 'grad_norm': 0.00022049406474201576, 'learning_rate': 3.241622574955909e-06, 'epoch': 1.42}\n",
      "{'loss': 0.0002, 'grad_norm': 0.03297421668535729, 'learning_rate': 3.2380952380952385e-06, 'epoch': 1.42}\n",
      "{'loss': 0.0004, 'grad_norm': 0.06763599693338382, 'learning_rate': 3.2345679012345682e-06, 'epoch': 1.42}\n",
      "{'loss': 0.4102, 'grad_norm': 33.65505307760262, 'learning_rate': 3.231040564373898e-06, 'epoch': 1.42}\n",
      "{'loss': 0.0055, 'grad_norm': 0.84653490141939, 'learning_rate': 3.2275132275132277e-06, 'epoch': 1.42}\n",
      "{'loss': 0.0, 'grad_norm': 2.3385771329576106e-06, 'learning_rate': 3.223985890652558e-06, 'epoch': 1.42}\n",
      "{'loss': 0.0001, 'grad_norm': 0.010680264708557253, 'learning_rate': 3.2204585537918875e-06, 'epoch': 1.42}\n",
      "{'loss': 0.0008, 'grad_norm': 0.14202907744194918, 'learning_rate': 3.2169312169312172e-06, 'epoch': 1.42}\n",
      "{'loss': 0.0, 'grad_norm': 0.0018984271255134146, 'learning_rate': 3.213403880070547e-06, 'epoch': 1.42}\n",
      "{'loss': 0.0, 'grad_norm': 0.00018546244547666205, 'learning_rate': 3.2098765432098767e-06, 'epoch': 1.42}\n",
      "{'loss': 0.0009, 'grad_norm': 0.09778856373826056, 'learning_rate': 3.206349206349207e-06, 'epoch': 1.42}\n",
      "{'loss': 0.0001, 'grad_norm': 0.008361516637371207, 'learning_rate': 3.2028218694885365e-06, 'epoch': 1.43}\n",
      "{'loss': 0.0, 'grad_norm': 0.0034338925300414908, 'learning_rate': 3.1992945326278662e-06, 'epoch': 1.43}\n",
      "{'loss': 0.22, 'grad_norm': 10.365980157066636, 'learning_rate': 3.195767195767196e-06, 'epoch': 1.43}\n",
      "{'loss': 0.0, 'grad_norm': 0.0029598183907788454, 'learning_rate': 3.1922398589065256e-06, 'epoch': 1.43}\n",
      "{'loss': 0.0, 'grad_norm': 0.0007502887601392175, 'learning_rate': 3.1887125220458558e-06, 'epoch': 1.43}\n",
      "{'loss': 0.0036, 'grad_norm': 0.3006645446868634, 'learning_rate': 3.1851851851851855e-06, 'epoch': 1.43}\n",
      "{'loss': 1.1592, 'grad_norm': 28.40788292886325, 'learning_rate': 3.181657848324515e-06, 'epoch': 1.43}\n",
      "{'loss': 0.0002, 'grad_norm': 0.01254880310030123, 'learning_rate': 3.178130511463845e-06, 'epoch': 1.43}\n",
      "{'loss': 0.0003, 'grad_norm': 0.03921559107119951, 'learning_rate': 3.1746031746031746e-06, 'epoch': 1.43}\n",
      "{'loss': 0.0, 'grad_norm': 0.00021084307975878167, 'learning_rate': 3.1710758377425048e-06, 'epoch': 1.43}\n",
      "{'loss': 0.0, 'grad_norm': 6.79170663969747e-06, 'learning_rate': 3.1675485008818345e-06, 'epoch': 1.43}\n",
      "{'loss': 0.0, 'grad_norm': 0.0021690081717920936, 'learning_rate': 3.164021164021164e-06, 'epoch': 1.43}\n",
      "{'loss': 0.0004, 'grad_norm': 0.04515570445539665, 'learning_rate': 3.160493827160494e-06, 'epoch': 1.43}\n",
      "{'loss': 0.0867, 'grad_norm': 4.513364866438624, 'learning_rate': 3.1569664902998236e-06, 'epoch': 1.43}\n",
      "{'loss': 0.0, 'grad_norm': 0.00019414769067375661, 'learning_rate': 3.1534391534391538e-06, 'epoch': 1.43}\n",
      "{'loss': 0.0, 'grad_norm': 0.0005440774922671246, 'learning_rate': 3.1499118165784835e-06, 'epoch': 1.43}\n",
      "{'loss': 0.4709, 'grad_norm': 14.214598466728228, 'learning_rate': 3.146384479717813e-06, 'epoch': 1.44}\n",
      "{'loss': 0.0006, 'grad_norm': 0.05760069586842938, 'learning_rate': 3.142857142857143e-06, 'epoch': 1.44}\n",
      "{'loss': 0.0001, 'grad_norm': 0.005998649960926443, 'learning_rate': 3.1393298059964726e-06, 'epoch': 1.44}\n",
      "{'loss': 0.0, 'grad_norm': 2.730710845159607e-06, 'learning_rate': 3.135802469135803e-06, 'epoch': 1.44}\n",
      "{'loss': 0.0, 'grad_norm': 0.0010912559269584436, 'learning_rate': 3.132275132275133e-06, 'epoch': 1.44}\n",
      "{'loss': 0.0002, 'grad_norm': 0.037670974335807246, 'learning_rate': 3.1287477954144626e-06, 'epoch': 1.44}\n",
      "{'loss': 0.0, 'grad_norm': 4.385223635370109e-05, 'learning_rate': 3.125220458553792e-06, 'epoch': 1.44}\n",
      "{'loss': 0.0, 'grad_norm': 0.00018964849228679295, 'learning_rate': 3.1216931216931216e-06, 'epoch': 1.44}\n",
      "{'loss': 0.002, 'grad_norm': 0.336699364849842, 'learning_rate': 3.118165784832452e-06, 'epoch': 1.44}\n",
      "{'loss': 0.0, 'grad_norm': 0.003402511629388774, 'learning_rate': 3.114638447971782e-06, 'epoch': 1.44}\n",
      "{'loss': 0.0, 'grad_norm': 0.00015035175650680532, 'learning_rate': 3.1111111111111116e-06, 'epoch': 1.44}\n",
      "{'loss': 0.0, 'grad_norm': 0.000584013986644131, 'learning_rate': 3.1075837742504413e-06, 'epoch': 1.44}\n",
      "{'loss': 0.0, 'grad_norm': 0.0008917964965648443, 'learning_rate': 3.104056437389771e-06, 'epoch': 1.44}\n",
      "{'loss': 0.0, 'grad_norm': 0.00012169705638181359, 'learning_rate': 3.100529100529101e-06, 'epoch': 1.44}\n",
      "{'loss': 0.4062, 'grad_norm': 20.56547548048139, 'learning_rate': 3.097001763668431e-06, 'epoch': 1.44}\n",
      "{'loss': 0.0, 'grad_norm': 0.0007914445416584449, 'learning_rate': 3.0934744268077606e-06, 'epoch': 1.45}\n",
      "{'loss': 0.0, 'grad_norm': 0.0015331599743789832, 'learning_rate': 3.0899470899470903e-06, 'epoch': 1.45}\n",
      "{'loss': 0.0, 'grad_norm': 2.579796857351788e-05, 'learning_rate': 3.08641975308642e-06, 'epoch': 1.45}\n",
      "{'loss': 0.0012, 'grad_norm': 0.14683450815368224, 'learning_rate': 3.08289241622575e-06, 'epoch': 1.45}\n",
      "{'loss': 0.0, 'grad_norm': 0.00021264980135498956, 'learning_rate': 3.07936507936508e-06, 'epoch': 1.45}\n",
      "{'loss': 0.0, 'grad_norm': 9.518565049082229e-06, 'learning_rate': 3.0758377425044096e-06, 'epoch': 1.45}\n",
      "{'loss': 0.0, 'grad_norm': 4.230346890509243e-05, 'learning_rate': 3.0723104056437393e-06, 'epoch': 1.45}\n",
      "{'loss': 0.0, 'grad_norm': 0.00046304857953536645, 'learning_rate': 3.068783068783069e-06, 'epoch': 1.45}\n",
      "{'loss': 0.0, 'grad_norm': 0.0005085454879077761, 'learning_rate': 3.065255731922399e-06, 'epoch': 1.45}\n",
      "{'loss': 0.0, 'grad_norm': 1.268607412948354e-06, 'learning_rate': 3.061728395061729e-06, 'epoch': 1.45}\n",
      "{'loss': 0.0, 'grad_norm': 2.5434231747057627e-05, 'learning_rate': 3.0582010582010585e-06, 'epoch': 1.45}\n",
      "{'loss': 0.0, 'grad_norm': 3.640798308521352e-05, 'learning_rate': 3.0546737213403883e-06, 'epoch': 1.45}\n",
      "{'loss': 0.0002, 'grad_norm': 0.014182125100745848, 'learning_rate': 3.051146384479718e-06, 'epoch': 1.45}\n",
      "{'loss': 0.0, 'grad_norm': 0.0004780171962416684, 'learning_rate': 3.047619047619048e-06, 'epoch': 1.45}\n",
      "{'loss': 0.0004, 'grad_norm': 0.057497058924710416, 'learning_rate': 3.044091710758378e-06, 'epoch': 1.45}\n",
      "{'loss': 0.0, 'grad_norm': 0.00027380243321475445, 'learning_rate': 3.0405643738977075e-06, 'epoch': 1.45}\n",
      "{'loss': 0.0113, 'grad_norm': 1.5693254547623599, 'learning_rate': 3.0370370370370372e-06, 'epoch': 1.46}\n",
      "{'loss': 0.0, 'grad_norm': 4.2533395304621724e-07, 'learning_rate': 3.033509700176367e-06, 'epoch': 1.46}\n",
      "{'loss': 0.0, 'grad_norm': 5.960675884014405e-05, 'learning_rate': 3.029982363315697e-06, 'epoch': 1.46}\n",
      "{'loss': 0.0008, 'grad_norm': 0.09047323480172968, 'learning_rate': 3.026455026455027e-06, 'epoch': 1.46}\n",
      "{'loss': 0.0, 'grad_norm': 0.00010061005824485249, 'learning_rate': 3.0229276895943565e-06, 'epoch': 1.46}\n",
      "{'loss': 0.0, 'grad_norm': 0.002647842949367073, 'learning_rate': 3.0194003527336862e-06, 'epoch': 1.46}\n",
      "{'loss': 0.0, 'grad_norm': 5.677861829539425e-06, 'learning_rate': 3.015873015873016e-06, 'epoch': 1.46}\n",
      "{'loss': 0.0004, 'grad_norm': 0.033870019700798455, 'learning_rate': 3.012345679012346e-06, 'epoch': 1.46}\n",
      "{'loss': 0.0, 'grad_norm': 0.00019282370665689812, 'learning_rate': 3.008818342151676e-06, 'epoch': 1.46}\n",
      "{'loss': 0.0, 'grad_norm': 3.991687667550919e-06, 'learning_rate': 3.0052910052910055e-06, 'epoch': 1.46}\n",
      "{'loss': 0.0, 'grad_norm': 0.0007687415469992445, 'learning_rate': 3.0017636684303352e-06, 'epoch': 1.46}\n",
      "{'loss': 0.0, 'grad_norm': 0.004339281857200881, 'learning_rate': 2.998236331569665e-06, 'epoch': 1.46}\n",
      "{'loss': 0.0, 'grad_norm': 0.003778253638370211, 'learning_rate': 2.9947089947089946e-06, 'epoch': 1.46}\n",
      "{'loss': 0.0, 'grad_norm': 2.6665129372781757e-05, 'learning_rate': 2.9911816578483248e-06, 'epoch': 1.46}\n",
      "{'loss': 0.0, 'grad_norm': 2.4053049880230055e-05, 'learning_rate': 2.9876543209876545e-06, 'epoch': 1.46}\n",
      "{'loss': 0.0004, 'grad_norm': 0.0335109827743625, 'learning_rate': 2.984126984126984e-06, 'epoch': 1.46}\n",
      "{'loss': 0.0002, 'grad_norm': 0.021064017039972516, 'learning_rate': 2.980599647266314e-06, 'epoch': 1.47}\n",
      "{'loss': 0.0, 'grad_norm': 0.00031259210292468087, 'learning_rate': 2.9770723104056436e-06, 'epoch': 1.47}\n",
      "{'loss': 0.0001, 'grad_norm': 0.007820347108368984, 'learning_rate': 2.9735449735449738e-06, 'epoch': 1.47}\n",
      "{'loss': 0.0, 'grad_norm': 0.00048684774660822007, 'learning_rate': 2.9700176366843035e-06, 'epoch': 1.47}\n",
      "{'loss': 0.0, 'grad_norm': 1.0899199930400296e-05, 'learning_rate': 2.966490299823633e-06, 'epoch': 1.47}\n",
      "{'loss': 0.0003, 'grad_norm': 0.04307775191205004, 'learning_rate': 2.962962962962963e-06, 'epoch': 1.47}\n",
      "{'loss': 0.0002, 'grad_norm': 0.016789151373202412, 'learning_rate': 2.9594356261022926e-06, 'epoch': 1.47}\n",
      "{'loss': 0.0003, 'grad_norm': 0.04038623724417128, 'learning_rate': 2.955908289241623e-06, 'epoch': 1.47}\n",
      "{'loss': 0.0004, 'grad_norm': 0.06544019130830676, 'learning_rate': 2.9523809523809525e-06, 'epoch': 1.47}\n",
      "{'loss': 0.0, 'grad_norm': 2.4109473271687417e-05, 'learning_rate': 2.948853615520282e-06, 'epoch': 1.47}\n",
      "{'loss': 0.0006, 'grad_norm': 0.09194715859722351, 'learning_rate': 2.945326278659612e-06, 'epoch': 1.47}\n",
      "{'loss': 0.0, 'grad_norm': 4.455464639757135e-05, 'learning_rate': 2.9417989417989416e-06, 'epoch': 1.47}\n",
      "{'loss': 0.0, 'grad_norm': 7.871233425165205e-05, 'learning_rate': 2.938271604938272e-06, 'epoch': 1.47}\n",
      "{'loss': 0.0, 'grad_norm': 0.0002670208656245146, 'learning_rate': 2.934744268077602e-06, 'epoch': 1.47}\n",
      "{'loss': 0.0, 'grad_norm': 0.00043199382360567756, 'learning_rate': 2.9312169312169316e-06, 'epoch': 1.47}\n",
      "{'loss': 0.0003, 'grad_norm': 0.025138885081248577, 'learning_rate': 2.9276895943562613e-06, 'epoch': 1.47}\n",
      "{'loss': 0.0, 'grad_norm': 1.5189428421868423e-05, 'learning_rate': 2.9241622574955906e-06, 'epoch': 1.48}\n",
      "{'loss': 0.0, 'grad_norm': 0.00028207875311963675, 'learning_rate': 2.920634920634921e-06, 'epoch': 1.48}\n",
      "{'loss': 0.0, 'grad_norm': 2.6145722187130794e-05, 'learning_rate': 2.917107583774251e-06, 'epoch': 1.48}\n",
      "{'loss': 0.1475, 'grad_norm': 6.030405568516995, 'learning_rate': 2.9135802469135806e-06, 'epoch': 1.48}\n",
      "{'loss': 0.0, 'grad_norm': 0.004270171718723482, 'learning_rate': 2.9100529100529103e-06, 'epoch': 1.48}\n",
      "{'loss': 0.0019, 'grad_norm': 0.23575723279272495, 'learning_rate': 2.90652557319224e-06, 'epoch': 1.48}\n",
      "{'loss': 0.0004, 'grad_norm': 0.03303715301903196, 'learning_rate': 2.90299823633157e-06, 'epoch': 1.48}\n",
      "{'loss': 1.3916, 'grad_norm': 28.716611896742364, 'learning_rate': 2.8994708994709e-06, 'epoch': 1.48}\n",
      "{'loss': 0.0, 'grad_norm': 0.0018450360297515298, 'learning_rate': 2.8959435626102296e-06, 'epoch': 1.48}\n",
      "{'loss': 0.0, 'grad_norm': 3.2658346743320666e-05, 'learning_rate': 2.8924162257495593e-06, 'epoch': 1.48}\n",
      "{'loss': 0.0185, 'grad_norm': 2.619884228951331, 'learning_rate': 2.888888888888889e-06, 'epoch': 1.48}\n",
      "{'loss': 0.0001, 'grad_norm': 0.009307744854057607, 'learning_rate': 2.885361552028219e-06, 'epoch': 1.48}\n",
      "{'loss': 0.0, 'grad_norm': 0.002606253632507715, 'learning_rate': 2.881834215167549e-06, 'epoch': 1.48}\n",
      "{'loss': 0.0001, 'grad_norm': 0.00820462237681794, 'learning_rate': 2.8783068783068786e-06, 'epoch': 1.48}\n",
      "{'loss': 0.0, 'grad_norm': 0.0008065107457961979, 'learning_rate': 2.8747795414462083e-06, 'epoch': 1.48}\n",
      "{'loss': 0.0, 'grad_norm': 2.6962570320025856e-05, 'learning_rate': 2.871252204585538e-06, 'epoch': 1.49}\n",
      "{'loss': 0.0002, 'grad_norm': 0.03192855397098482, 'learning_rate': 2.867724867724868e-06, 'epoch': 1.49}\n",
      "{'loss': 0.0, 'grad_norm': 0.00022173301984594187, 'learning_rate': 2.864197530864198e-06, 'epoch': 1.49}\n",
      "{'loss': 0.0, 'grad_norm': 0.00012512698748549803, 'learning_rate': 2.8606701940035275e-06, 'epoch': 1.49}\n",
      "{'loss': 0.0, 'grad_norm': 0.00020959621408115064, 'learning_rate': 2.8571428571428573e-06, 'epoch': 1.49}\n",
      "{'loss': 0.0, 'grad_norm': 8.074147105614219e-07, 'learning_rate': 2.853615520282187e-06, 'epoch': 1.49}\n",
      "{'loss': 0.0, 'grad_norm': 0.003864403065844997, 'learning_rate': 2.850088183421517e-06, 'epoch': 1.49}\n",
      "{'loss': 0.0, 'grad_norm': 5.02304511567028e-06, 'learning_rate': 2.846560846560847e-06, 'epoch': 1.49}\n",
      "{'loss': 0.0, 'grad_norm': 0.0041753203404013076, 'learning_rate': 2.8430335097001765e-06, 'epoch': 1.49}\n",
      "{'loss': 0.0, 'grad_norm': 0.0011662612727624685, 'learning_rate': 2.8395061728395062e-06, 'epoch': 1.49}\n",
      "{'loss': 0.2896, 'grad_norm': 12.263546674040393, 'learning_rate': 2.835978835978836e-06, 'epoch': 1.49}\n",
      "{'loss': 0.0, 'grad_norm': 0.0012641722437361782, 'learning_rate': 2.832451499118166e-06, 'epoch': 1.49}\n",
      "{'loss': 0.0, 'grad_norm': 0.00013542428452936102, 'learning_rate': 2.828924162257496e-06, 'epoch': 1.49}\n",
      "{'loss': 0.0003, 'grad_norm': 0.04234585177696919, 'learning_rate': 2.8253968253968255e-06, 'epoch': 1.49}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 75%|███████▍  | 2352/3150 [09:50<02:43,  4.89it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.0, 'grad_norm': 0.001067494846590443, 'learning_rate': 2.8218694885361552e-06, 'epoch': 1.49}\n",
      "{'loss': 0.0, 'grad_norm': 0.0019456256210602489, 'learning_rate': 2.818342151675485e-06, 'epoch': 1.49}\n",
      "{'loss': 0.003, 'grad_norm': 0.3920454900412361, 'learning_rate': 2.814814814814815e-06, 'epoch': 1.5}\n",
      "{'loss': 0.0, 'grad_norm': 1.4884702153316084e-05, 'learning_rate': 2.811287477954145e-06, 'epoch': 1.5}\n",
      "{'loss': 0.0, 'grad_norm': 0.000446312017993157, 'learning_rate': 2.8077601410934745e-06, 'epoch': 1.5}\n",
      "{'loss': 0.0, 'grad_norm': 0.0011505404281340935, 'learning_rate': 2.8042328042328042e-06, 'epoch': 1.5}\n",
      "{'loss': 1.3574, 'grad_norm': 22.403333742837667, 'learning_rate': 2.800705467372134e-06, 'epoch': 1.5}\n",
      "{'loss': 0.0055, 'grad_norm': 0.5809690268068924, 'learning_rate': 2.797178130511464e-06, 'epoch': 1.5}\n",
      "{'loss': 0.0, 'grad_norm': 0.0002727443025917454, 'learning_rate': 2.7936507936507938e-06, 'epoch': 1.5}\n",
      "{'loss': 0.0, 'grad_norm': 0.00023945617647247515, 'learning_rate': 2.7901234567901235e-06, 'epoch': 1.5}\n",
      "{'loss': 0.0, 'grad_norm': 3.092883570111503e-06, 'learning_rate': 2.786596119929453e-06, 'epoch': 1.5}\n",
      "{'loss': 0.0, 'grad_norm': 0.0027290091688050274, 'learning_rate': 2.783068783068783e-06, 'epoch': 1.5}\n",
      "{'loss': 0.0018, 'grad_norm': 0.19708339848090528, 'learning_rate': 2.7795414462081135e-06, 'epoch': 1.5}\n",
      "{'loss': 0.0002, 'grad_norm': 0.021264053970375216, 'learning_rate': 2.7760141093474428e-06, 'epoch': 1.5}\n",
      "{'loss': 0.0, 'grad_norm': 0.0003340116870431908, 'learning_rate': 2.7724867724867725e-06, 'epoch': 1.5}\n",
      "{'loss': 0.0, 'grad_norm': 9.169761502275282e-05, 'learning_rate': 2.768959435626102e-06, 'epoch': 1.5}\n",
      "{'loss': 0.0, 'grad_norm': 0.00020704895749528286, 'learning_rate': 2.765432098765432e-06, 'epoch': 1.5}\n",
      "{'loss': 0.0, 'grad_norm': 0.0005517328919704107, 'learning_rate': 2.7619047619047625e-06, 'epoch': 1.5}\n",
      "{'loss': 0.0, 'grad_norm': 4.905203768095536e-06, 'learning_rate': 2.758377425044092e-06, 'epoch': 1.51}\n",
      "{'loss': 0.0, 'grad_norm': 0.0022497526815364044, 'learning_rate': 2.754850088183422e-06, 'epoch': 1.51}\n",
      "{'loss': 0.0, 'grad_norm': 0.00018939158044833422, 'learning_rate': 2.7513227513227516e-06, 'epoch': 1.51}\n",
      "{'loss': 0.0001, 'grad_norm': 0.01587002183228159, 'learning_rate': 2.747795414462081e-06, 'epoch': 1.51}\n",
      "{'loss': 0.0002, 'grad_norm': 0.02214727160246167, 'learning_rate': 2.7442680776014115e-06, 'epoch': 1.51}\n",
      "{'loss': 0.0, 'grad_norm': 2.6595077596027427e-06, 'learning_rate': 2.740740740740741e-06, 'epoch': 1.51}\n",
      "{'loss': 0.0, 'grad_norm': 0.0004887874233931975, 'learning_rate': 2.737213403880071e-06, 'epoch': 1.51}\n",
      "{'loss': 0.0, 'grad_norm': 0.0009314845154158948, 'learning_rate': 2.7336860670194006e-06, 'epoch': 1.51}\n",
      "{'loss': 0.0039, 'grad_norm': 0.456949715799706, 'learning_rate': 2.7301587301587303e-06, 'epoch': 1.51}\n",
      "{'loss': 0.0004, 'grad_norm': 0.04149736465063521, 'learning_rate': 2.7266313932980604e-06, 'epoch': 1.51}\n",
      "{'loss': 0.0001, 'grad_norm': 0.0075334314521293255, 'learning_rate': 2.72310405643739e-06, 'epoch': 1.51}\n",
      "{'loss': 0.0006, 'grad_norm': 0.05196251067374117, 'learning_rate': 2.71957671957672e-06, 'epoch': 1.51}\n",
      "{'loss': 0.0, 'grad_norm': 0.00016806805618107813, 'learning_rate': 2.7160493827160496e-06, 'epoch': 1.51}\n",
      "{'loss': 0.012, 'grad_norm': 1.3247870292088497, 'learning_rate': 2.7125220458553793e-06, 'epoch': 1.51}\n",
      "{'loss': 0.0, 'grad_norm': 4.7602304609279785e-05, 'learning_rate': 2.7089947089947094e-06, 'epoch': 1.51}\n",
      "{'loss': 0.0001, 'grad_norm': 0.011422876402439903, 'learning_rate': 2.705467372134039e-06, 'epoch': 1.51}\n",
      "{'loss': 0.0337, 'grad_norm': 4.157785461865417, 'learning_rate': 2.701940035273369e-06, 'epoch': 1.52}\n",
      "{'loss': 0.0, 'grad_norm': 0.00011855224123203816, 'learning_rate': 2.6984126984126986e-06, 'epoch': 1.52}\n",
      "{'loss': 0.0, 'grad_norm': 0.0006666358701915959, 'learning_rate': 2.6948853615520283e-06, 'epoch': 1.52}\n",
      "{'loss': 0.0, 'grad_norm': 6.74332190375177e-05, 'learning_rate': 2.6913580246913584e-06, 'epoch': 1.52}\n",
      "{'loss': 0.0, 'grad_norm': 1.1246792080297375e-05, 'learning_rate': 2.687830687830688e-06, 'epoch': 1.52}\n",
      "{'loss': 0.0001, 'grad_norm': 0.008914679399603992, 'learning_rate': 2.684303350970018e-06, 'epoch': 1.52}\n",
      "{'loss': 0.0, 'grad_norm': 0.0018100833993063058, 'learning_rate': 2.6807760141093476e-06, 'epoch': 1.52}\n",
      "{'loss': 0.0, 'grad_norm': 0.0014763428249026517, 'learning_rate': 2.6772486772486773e-06, 'epoch': 1.52}\n",
      "{'loss': 0.0, 'grad_norm': 0.0002346836440057346, 'learning_rate': 2.6737213403880074e-06, 'epoch': 1.52}\n",
      "{'loss': 0.0, 'grad_norm': 0.0006520104986758714, 'learning_rate': 2.670194003527337e-06, 'epoch': 1.52}\n",
      "{'loss': 0.0005, 'grad_norm': 0.07942014283462669, 'learning_rate': 2.666666666666667e-06, 'epoch': 1.52}\n",
      "{'loss': 0.0, 'grad_norm': 7.96580064075826e-05, 'learning_rate': 2.6631393298059965e-06, 'epoch': 1.52}\n",
      "{'loss': 0.0, 'grad_norm': 0.00033914629099019594, 'learning_rate': 2.6596119929453263e-06, 'epoch': 1.52}\n",
      "{'loss': 0.0, 'grad_norm': 0.0019337763957436745, 'learning_rate': 2.6560846560846564e-06, 'epoch': 1.52}\n",
      "{'loss': 0.0, 'grad_norm': 0.0020308946320990117, 'learning_rate': 2.652557319223986e-06, 'epoch': 1.52}\n",
      "{'loss': 0.0005, 'grad_norm': 0.047810434650554505, 'learning_rate': 2.649029982363316e-06, 'epoch': 1.53}\n",
      "{'loss': 0.0, 'grad_norm': 1.3382338929850441e-05, 'learning_rate': 2.6455026455026455e-06, 'epoch': 1.53}\n",
      "{'loss': 0.0, 'grad_norm': 0.001059326667840751, 'learning_rate': 2.6419753086419752e-06, 'epoch': 1.53}\n",
      "{'loss': 0.0013, 'grad_norm': 0.1927535072853628, 'learning_rate': 2.6384479717813054e-06, 'epoch': 1.53}\n",
      "{'loss': 0.0057, 'grad_norm': 0.6724126350279447, 'learning_rate': 2.634920634920635e-06, 'epoch': 1.53}\n",
      "{'loss': 0.0, 'grad_norm': 4.6433694391304115e-05, 'learning_rate': 2.631393298059965e-06, 'epoch': 1.53}\n",
      "{'loss': 0.0002, 'grad_norm': 0.01742610329210908, 'learning_rate': 2.6278659611992945e-06, 'epoch': 1.53}\n",
      "{'loss': 0.0, 'grad_norm': 8.420819722053794e-07, 'learning_rate': 2.6243386243386242e-06, 'epoch': 1.53}\n",
      "{'loss': 0.0, 'grad_norm': 5.871633353867689e-06, 'learning_rate': 2.6208112874779544e-06, 'epoch': 1.53}\n",
      "{'loss': 0.0, 'grad_norm': 0.0025541181373878357, 'learning_rate': 2.617283950617284e-06, 'epoch': 1.53}\n",
      "{'loss': 0.0, 'grad_norm': 0.0003414959369681774, 'learning_rate': 2.613756613756614e-06, 'epoch': 1.53}\n",
      "{'loss': 0.0, 'grad_norm': 0.0005486800737155373, 'learning_rate': 2.6102292768959435e-06, 'epoch': 1.53}\n",
      "{'loss': 0.0, 'grad_norm': 0.00048733805887308764, 'learning_rate': 2.6067019400352732e-06, 'epoch': 1.53}\n",
      "{'loss': 0.0, 'grad_norm': 0.00021257150687811497, 'learning_rate': 2.6031746031746038e-06, 'epoch': 1.53}\n",
      "{'loss': 0.0, 'grad_norm': 3.012839670184627e-05, 'learning_rate': 2.599647266313933e-06, 'epoch': 1.53}\n",
      "{'loss': 0.0, 'grad_norm': 7.825763853852161e-05, 'learning_rate': 2.5961199294532628e-06, 'epoch': 1.53}\n",
      "{'loss': 0.0, 'grad_norm': 0.0015861615187050622, 'learning_rate': 2.5925925925925925e-06, 'epoch': 1.54}\n",
      "{'loss': 0.0, 'grad_norm': 0.0035497253105659667, 'learning_rate': 2.589065255731922e-06, 'epoch': 1.54}\n",
      "{'loss': 0.0, 'grad_norm': 5.454199676015437e-06, 'learning_rate': 2.5855379188712528e-06, 'epoch': 1.54}\n",
      "{'loss': 0.0, 'grad_norm': 0.0010251732201766665, 'learning_rate': 2.5820105820105825e-06, 'epoch': 1.54}\n",
      "{'loss': 0.0, 'grad_norm': 2.9374932712049338e-06, 'learning_rate': 2.578483245149912e-06, 'epoch': 1.54}\n",
      "{'loss': 0.0, 'grad_norm': 0.00018164389097808232, 'learning_rate': 2.574955908289242e-06, 'epoch': 1.54}\n",
      "{'loss': 0.0378, 'grad_norm': 5.258270289087988, 'learning_rate': 2.571428571428571e-06, 'epoch': 1.54}\n",
      "{'loss': 0.0, 'grad_norm': 3.746790352667127e-06, 'learning_rate': 2.5679012345679018e-06, 'epoch': 1.54}\n",
      "{'loss': 0.0004, 'grad_norm': 0.05434644523792538, 'learning_rate': 2.5643738977072315e-06, 'epoch': 1.54}\n",
      "{'loss': 0.0437, 'grad_norm': 4.886745486350221, 'learning_rate': 2.560846560846561e-06, 'epoch': 1.54}\n",
      "{'loss': 0.0, 'grad_norm': 1.3493378921663138e-05, 'learning_rate': 2.557319223985891e-06, 'epoch': 1.54}\n",
      "{'loss': 0.0, 'grad_norm': 0.0003024190366928949, 'learning_rate': 2.5537918871252206e-06, 'epoch': 1.54}\n",
      "{'loss': 0.0, 'grad_norm': 0.0013020777938924865, 'learning_rate': 2.5502645502645507e-06, 'epoch': 1.54}\n",
      "{'loss': 0.0, 'grad_norm': 1.2649586763230201e-05, 'learning_rate': 2.5467372134038805e-06, 'epoch': 1.54}\n",
      "{'loss': 0.0, 'grad_norm': 0.00010306164910738283, 'learning_rate': 2.54320987654321e-06, 'epoch': 1.54}\n",
      "{'loss': 0.0009, 'grad_norm': 0.09340646007325334, 'learning_rate': 2.53968253968254e-06, 'epoch': 1.54}\n",
      "{'loss': 0.0, 'grad_norm': 2.7993148063132336e-05, 'learning_rate': 2.5361552028218696e-06, 'epoch': 1.55}\n",
      "{'loss': 0.0, 'grad_norm': 0.0016998721030410315, 'learning_rate': 2.5326278659611997e-06, 'epoch': 1.55}\n",
      "{'loss': 0.0497, 'grad_norm': 5.279654538383902, 'learning_rate': 2.5291005291005294e-06, 'epoch': 1.55}\n",
      "{'loss': 0.0, 'grad_norm': 0.0007020469771886021, 'learning_rate': 2.525573192239859e-06, 'epoch': 1.55}\n",
      "{'loss': 0.0015, 'grad_norm': 0.16469183629278997, 'learning_rate': 2.522045855379189e-06, 'epoch': 1.55}\n",
      "{'loss': 0.0, 'grad_norm': 0.0017455491560409212, 'learning_rate': 2.5185185185185186e-06, 'epoch': 1.55}\n",
      "{'loss': 0.0, 'grad_norm': 0.0001255097483252382, 'learning_rate': 2.5149911816578487e-06, 'epoch': 1.55}\n",
      "{'loss': 0.0004, 'grad_norm': 0.05164729320566616, 'learning_rate': 2.5114638447971784e-06, 'epoch': 1.55}\n",
      "{'loss': 0.0, 'grad_norm': 1.2563676154934973e-05, 'learning_rate': 2.507936507936508e-06, 'epoch': 1.55}\n",
      "{'loss': 0.0, 'grad_norm': 4.53464280205773e-06, 'learning_rate': 2.504409171075838e-06, 'epoch': 1.55}\n",
      "{'loss': 0.0, 'grad_norm': 0.0057404517739273, 'learning_rate': 2.5008818342151676e-06, 'epoch': 1.55}\n",
      "{'loss': 0.0026, 'grad_norm': 0.21186022799823975, 'learning_rate': 2.4973544973544973e-06, 'epoch': 1.55}\n",
      "{'loss': 0.0, 'grad_norm': 6.826701554129248e-05, 'learning_rate': 2.4938271604938274e-06, 'epoch': 1.55}\n",
      "{'loss': 0.0001, 'grad_norm': 0.009075660259099639, 'learning_rate': 2.490299823633157e-06, 'epoch': 1.55}\n",
      "{'loss': 0.0, 'grad_norm': 0.0019486354636758455, 'learning_rate': 2.486772486772487e-06, 'epoch': 1.55}\n",
      "{'loss': 0.0, 'grad_norm': 9.951032444266097e-06, 'learning_rate': 2.483245149911817e-06, 'epoch': 1.55}\n",
      "{'loss': 0.0001, 'grad_norm': 0.009079078064711983, 'learning_rate': 2.4797178130511467e-06, 'epoch': 1.56}\n",
      "{'loss': 0.0, 'grad_norm': 0.0004221730130390719, 'learning_rate': 2.4761904761904764e-06, 'epoch': 1.56}\n",
      "{'loss': 0.0001, 'grad_norm': 0.006272460906245377, 'learning_rate': 2.472663139329806e-06, 'epoch': 1.56}\n",
      "{'loss': 0.0, 'grad_norm': 1.3964351430734386e-05, 'learning_rate': 2.469135802469136e-06, 'epoch': 1.56}\n",
      "{'loss': 0.0, 'grad_norm': 0.003008231359409712, 'learning_rate': 2.465608465608466e-06, 'epoch': 1.56}\n",
      "{'loss': 0.0, 'grad_norm': 0.0006982613082675657, 'learning_rate': 2.4620811287477957e-06, 'epoch': 1.56}\n",
      "{'loss': 0.0016, 'grad_norm': 0.15047590513042705, 'learning_rate': 2.4585537918871254e-06, 'epoch': 1.56}\n",
      "{'loss': 0.0, 'grad_norm': 0.0017598526613240132, 'learning_rate': 2.455026455026455e-06, 'epoch': 1.56}\n",
      "{'loss': 0.0378, 'grad_norm': 4.735509352553851, 'learning_rate': 2.451499118165785e-06, 'epoch': 1.56}\n",
      "{'loss': 0.0, 'grad_norm': 2.2110835361017898e-05, 'learning_rate': 2.447971781305115e-06, 'epoch': 1.56}\n",
      "{'loss': 0.0832, 'grad_norm': 9.661291886341388, 'learning_rate': 2.4444444444444447e-06, 'epoch': 1.56}\n",
      "{'loss': 0.0, 'grad_norm': 2.2900871949808142e-07, 'learning_rate': 2.4409171075837744e-06, 'epoch': 1.56}\n",
      "{'loss': 0.2218, 'grad_norm': 19.03221932183562, 'learning_rate': 2.437389770723104e-06, 'epoch': 1.56}\n",
      "{'loss': 0.0, 'grad_norm': 0.00018110558184389534, 'learning_rate': 2.433862433862434e-06, 'epoch': 1.56}\n",
      "{'loss': 0.0, 'grad_norm': 9.911388451000248e-06, 'learning_rate': 2.430335097001764e-06, 'epoch': 1.56}\n",
      "{'loss': 0.0, 'grad_norm': 0.0006807174023554377, 'learning_rate': 2.4268077601410937e-06, 'epoch': 1.57}\n",
      "{'loss': 0.0855, 'grad_norm': 9.76332199406858, 'learning_rate': 2.4232804232804234e-06, 'epoch': 1.57}\n",
      "{'loss': 0.0, 'grad_norm': 7.960801410277151e-05, 'learning_rate': 2.419753086419753e-06, 'epoch': 1.57}\n",
      "{'loss': 0.0001, 'grad_norm': 0.009860339410583015, 'learning_rate': 2.416225749559083e-06, 'epoch': 1.57}\n",
      "{'loss': 0.0002, 'grad_norm': 0.020422773362789217, 'learning_rate': 2.412698412698413e-06, 'epoch': 1.57}\n",
      "{'loss': 0.0, 'grad_norm': 0.0015278887922665087, 'learning_rate': 2.4091710758377426e-06, 'epoch': 1.57}\n",
      "{'loss': 0.0027, 'grad_norm': 0.27699121716326275, 'learning_rate': 2.4056437389770728e-06, 'epoch': 1.57}\n",
      "{'loss': 0.0068, 'grad_norm': 0.6784253075135332, 'learning_rate': 2.4021164021164025e-06, 'epoch': 1.57}\n",
      "{'loss': 0.0, 'grad_norm': 1.5185064641428105e-06, 'learning_rate': 2.3985890652557318e-06, 'epoch': 1.57}\n",
      "{'loss': 0.0, 'grad_norm': 0.0006195021077369671, 'learning_rate': 2.395061728395062e-06, 'epoch': 1.57}\n",
      "{'loss': 0.0, 'grad_norm': 1.526432233449919e-05, 'learning_rate': 2.3915343915343916e-06, 'epoch': 1.57}\n",
      "{'loss': 0.4189, 'grad_norm': 12.23288512140757, 'learning_rate': 2.3880070546737218e-06, 'epoch': 1.57}\n",
      "{'loss': 0.0021, 'grad_norm': 0.16136282485313377, 'learning_rate': 2.3844797178130515e-06, 'epoch': 1.57}\n",
      "{'loss': 0.0066, 'grad_norm': 0.7360621207107431, 'learning_rate': 2.380952380952381e-06, 'epoch': 1.57}\n",
      "{'loss': 0.0, 'grad_norm': 1.3612954487848667e-07, 'learning_rate': 2.377425044091711e-06, 'epoch': 1.57}\n",
      "{'loss': 0.0, 'grad_norm': 0.004567785192486753, 'learning_rate': 2.3738977072310406e-06, 'epoch': 1.57}\n",
      "{'loss': 0.0, 'grad_norm': 0.0058301963441730244, 'learning_rate': 2.3703703703703707e-06, 'epoch': 1.58}\n",
      "{'loss': 0.0, 'grad_norm': 0.00019829644772976083, 'learning_rate': 2.3668430335097005e-06, 'epoch': 1.58}\n",
      "{'loss': 0.0001, 'grad_norm': 0.009108424323222997, 'learning_rate': 2.36331569664903e-06, 'epoch': 1.58}\n",
      "{'loss': 0.0003, 'grad_norm': 0.03491728750922038, 'learning_rate': 2.35978835978836e-06, 'epoch': 1.58}\n",
      "{'loss': 0.2131, 'grad_norm': 13.93147072152986, 'learning_rate': 2.3562610229276896e-06, 'epoch': 1.58}\n",
      "{'loss': 0.0, 'grad_norm': 4.937264276245653e-05, 'learning_rate': 2.3527336860670197e-06, 'epoch': 1.58}\n",
      "{'loss': 0.0, 'grad_norm': 0.00254040866365097, 'learning_rate': 2.3492063492063494e-06, 'epoch': 1.58}\n",
      "{'loss': 0.0665, 'grad_norm': 8.149436940594759, 'learning_rate': 2.345679012345679e-06, 'epoch': 1.58}\n",
      "{'loss': 0.0, 'grad_norm': 0.0006179026059899977, 'learning_rate': 2.342151675485009e-06, 'epoch': 1.58}\n",
      "{'loss': 0.0, 'grad_norm': 2.573927506769347e-06, 'learning_rate': 2.3386243386243386e-06, 'epoch': 1.58}\n",
      "{'loss': 0.079, 'grad_norm': 6.8982375948703885, 'learning_rate': 2.3350970017636687e-06, 'epoch': 1.58}\n",
      "{'loss': 0.0, 'grad_norm': 6.81762603038695e-06, 'learning_rate': 2.3315696649029984e-06, 'epoch': 1.58}\n",
      "{'loss': 0.0, 'grad_norm': 9.516661332320596e-05, 'learning_rate': 2.328042328042328e-06, 'epoch': 1.58}\n",
      "{'loss': 0.0, 'grad_norm': 0.0011146373795598834, 'learning_rate': 2.3245149911816583e-06, 'epoch': 1.58}\n",
      "{'loss': 0.0, 'grad_norm': 0.00019821306624122147, 'learning_rate': 2.3209876543209876e-06, 'epoch': 1.58}\n",
      "{'loss': 0.0, 'grad_norm': 5.696238377050539e-05, 'learning_rate': 2.3174603174603177e-06, 'epoch': 1.58}\n",
      "{'loss': 0.0, 'grad_norm': 0.0044571831950861555, 'learning_rate': 2.3139329805996474e-06, 'epoch': 1.59}\n",
      "{'loss': 0.0, 'grad_norm': 0.00016651671000516064, 'learning_rate': 2.310405643738977e-06, 'epoch': 1.59}\n",
      "{'loss': 0.0004, 'grad_norm': 0.045650522820642775, 'learning_rate': 2.3068783068783073e-06, 'epoch': 1.59}\n",
      "{'loss': 0.0, 'grad_norm': 8.992574110926768e-05, 'learning_rate': 2.303350970017637e-06, 'epoch': 1.59}\n",
      "{'loss': 0.0014, 'grad_norm': 0.18757386540760326, 'learning_rate': 2.2998236331569667e-06, 'epoch': 1.59}\n",
      "{'loss': 0.0, 'grad_norm': 0.003666851703367774, 'learning_rate': 2.2962962962962964e-06, 'epoch': 1.59}\n",
      "{'loss': 0.0, 'grad_norm': 9.986192891361487e-07, 'learning_rate': 2.292768959435626e-06, 'epoch': 1.59}\n",
      "{'loss': 0.0, 'grad_norm': 0.0017636870281781995, 'learning_rate': 2.2892416225749563e-06, 'epoch': 1.59}\n",
      "{'loss': 0.3113, 'grad_norm': 12.370497992443743, 'learning_rate': 2.285714285714286e-06, 'epoch': 1.59}\n",
      "{'loss': 0.0001, 'grad_norm': 0.009395849880482816, 'learning_rate': 2.2821869488536157e-06, 'epoch': 1.59}\n",
      "{'loss': 0.0, 'grad_norm': 0.00022280744807339541, 'learning_rate': 2.2786596119929454e-06, 'epoch': 1.59}\n",
      "{'loss': 0.0, 'grad_norm': 4.634458052416317e-05, 'learning_rate': 2.275132275132275e-06, 'epoch': 1.59}\n",
      "{'loss': 0.0003, 'grad_norm': 0.035554412681893834, 'learning_rate': 2.2716049382716052e-06, 'epoch': 1.59}\n",
      "{'loss': 0.0003, 'grad_norm': 0.03382290182029125, 'learning_rate': 2.268077601410935e-06, 'epoch': 1.59}\n",
      "{'loss': 0.0, 'grad_norm': 1.4042366028248359e-05, 'learning_rate': 2.2645502645502647e-06, 'epoch': 1.59}\n",
      "{'loss': 0.0, 'grad_norm': 0.0017629798918001836, 'learning_rate': 2.2610229276895944e-06, 'epoch': 1.59}\n",
      "{'loss': 0.0009, 'grad_norm': 0.12686340389439, 'learning_rate': 2.257495590828924e-06, 'epoch': 1.6}\n",
      "{'loss': 0.0, 'grad_norm': 0.0012516429525915446, 'learning_rate': 2.2539682539682542e-06, 'epoch': 1.6}\n",
      "{'loss': 0.0, 'grad_norm': 4.941929482878722e-06, 'learning_rate': 2.250440917107584e-06, 'epoch': 1.6}\n",
      "{'loss': 0.0002, 'grad_norm': 0.02211244410183765, 'learning_rate': 2.2469135802469137e-06, 'epoch': 1.6}\n",
      "{'loss': 0.0, 'grad_norm': 0.0028809287277052798, 'learning_rate': 2.2433862433862434e-06, 'epoch': 1.6}\n",
      "{'loss': 0.0002, 'grad_norm': 0.021387003516779163, 'learning_rate': 2.239858906525573e-06, 'epoch': 1.6}\n",
      "{'loss': 0.0001, 'grad_norm': 0.011766148491239345, 'learning_rate': 2.2363315696649032e-06, 'epoch': 1.6}\n",
      "{'loss': 0.0, 'grad_norm': 0.004185362256708367, 'learning_rate': 2.232804232804233e-06, 'epoch': 1.6}\n",
      "{'loss': 0.0, 'grad_norm': 0.001337265347505097, 'learning_rate': 2.229276895943563e-06, 'epoch': 1.6}\n",
      "{'loss': 0.0001, 'grad_norm': 0.008319218491858049, 'learning_rate': 2.2257495590828928e-06, 'epoch': 1.6}\n",
      "{'loss': 0.0, 'grad_norm': 0.0006526416941880093, 'learning_rate': 2.222222222222222e-06, 'epoch': 1.6}\n",
      "{'loss': 0.0014, 'grad_norm': 0.19622887215202572, 'learning_rate': 2.218694885361552e-06, 'epoch': 1.6}\n",
      "{'loss': 0.0, 'grad_norm': 0.00014998835285247455, 'learning_rate': 2.215167548500882e-06, 'epoch': 1.6}\n",
      "{'loss': 0.0, 'grad_norm': 0.0025549833773884857, 'learning_rate': 2.211640211640212e-06, 'epoch': 1.6}\n",
      "{'loss': 0.0, 'grad_norm': 0.00022922070291820469, 'learning_rate': 2.2081128747795418e-06, 'epoch': 1.6}\n",
      "{'loss': 0.0, 'grad_norm': 0.0015571720280714225, 'learning_rate': 2.2045855379188715e-06, 'epoch': 1.61}\n",
      "{'loss': 0.0004, 'grad_norm': 0.03141945662965893, 'learning_rate': 2.201058201058201e-06, 'epoch': 1.61}\n",
      "{'loss': 0.0034, 'grad_norm': 0.49694154041918503, 'learning_rate': 2.197530864197531e-06, 'epoch': 1.61}\n",
      "{'loss': 0.0, 'grad_norm': 8.078773622834898e-05, 'learning_rate': 2.194003527336861e-06, 'epoch': 1.61}\n",
      "{'loss': 0.0, 'grad_norm': 7.89548877882176e-05, 'learning_rate': 2.1904761904761908e-06, 'epoch': 1.61}\n",
      "{'loss': 0.0, 'grad_norm': 0.00021188897080023068, 'learning_rate': 2.1869488536155205e-06, 'epoch': 1.61}\n",
      "{'loss': 0.0, 'grad_norm': 5.324585872087369e-05, 'learning_rate': 2.18342151675485e-06, 'epoch': 1.61}\n",
      "{'loss': 0.0, 'grad_norm': 0.0005499360330455849, 'learning_rate': 2.17989417989418e-06, 'epoch': 1.61}\n",
      "{'loss': 0.1146, 'grad_norm': 11.228714876634891, 'learning_rate': 2.17636684303351e-06, 'epoch': 1.61}\n",
      "{'loss': 0.0743, 'grad_norm': 10.108058743522419, 'learning_rate': 2.1728395061728397e-06, 'epoch': 1.61}\n",
      "{'loss': 0.0, 'grad_norm': 0.001260467460238435, 'learning_rate': 2.1693121693121695e-06, 'epoch': 1.61}\n",
      "{'loss': 0.0, 'grad_norm': 6.0915165081880466e-05, 'learning_rate': 2.165784832451499e-06, 'epoch': 1.61}\n",
      "{'loss': 0.0019, 'grad_norm': 0.27940758598306037, 'learning_rate': 2.162257495590829e-06, 'epoch': 1.61}\n",
      "{'loss': 0.0, 'grad_norm': 0.00033041752980255497, 'learning_rate': 2.158730158730159e-06, 'epoch': 1.61}\n",
      "{'loss': 0.0, 'grad_norm': 0.0003891311864870363, 'learning_rate': 2.1552028218694887e-06, 'epoch': 1.61}\n",
      "{'loss': 0.0001, 'grad_norm': 0.006958191329644049, 'learning_rate': 2.1516754850088184e-06, 'epoch': 1.61}\n",
      "{'loss': 0.0, 'grad_norm': 0.0012091214364195787, 'learning_rate': 2.148148148148148e-06, 'epoch': 1.62}\n",
      "{'loss': 0.0001, 'grad_norm': 0.012349786497128241, 'learning_rate': 2.144620811287478e-06, 'epoch': 1.62}\n",
      "{'loss': 0.0, 'grad_norm': 0.0060556898325676446, 'learning_rate': 2.141093474426808e-06, 'epoch': 1.62}\n",
      "{'loss': 0.0, 'grad_norm': 0.0056370339226608, 'learning_rate': 2.1375661375661377e-06, 'epoch': 1.62}\n",
      "{'loss': 0.0001, 'grad_norm': 0.006756552555690485, 'learning_rate': 2.1340388007054674e-06, 'epoch': 1.62}\n",
      "{'loss': 0.0002, 'grad_norm': 0.019573662573691666, 'learning_rate': 2.1305114638447976e-06, 'epoch': 1.62}\n",
      "{'loss': 0.0, 'grad_norm': 0.00032645007410282234, 'learning_rate': 2.1269841269841273e-06, 'epoch': 1.62}\n",
      "{'loss': 0.0, 'grad_norm': 0.00010894146720618194, 'learning_rate': 2.123456790123457e-06, 'epoch': 1.62}\n",
      "{'loss': 0.0, 'grad_norm': 0.0003685475637354514, 'learning_rate': 2.1199294532627867e-06, 'epoch': 1.62}\n",
      "{'loss': 0.0, 'grad_norm': 3.9179128675984593e-05, 'learning_rate': 2.1164021164021164e-06, 'epoch': 1.62}\n",
      "{'loss': 0.2817, 'grad_norm': 20.671453516347505, 'learning_rate': 2.1128747795414466e-06, 'epoch': 1.62}\n",
      "{'loss': 0.366, 'grad_norm': 29.76498420406046, 'learning_rate': 2.1093474426807763e-06, 'epoch': 1.62}\n",
      "{'loss': 0.003, 'grad_norm': 0.32127630793986706, 'learning_rate': 2.105820105820106e-06, 'epoch': 1.62}\n",
      "{'loss': 0.0, 'grad_norm': 5.994035015417779e-05, 'learning_rate': 2.1022927689594357e-06, 'epoch': 1.62}\n",
      "{'loss': 0.0, 'grad_norm': 0.0015317977164872652, 'learning_rate': 2.0987654320987654e-06, 'epoch': 1.62}\n",
      "{'loss': 0.8711, 'grad_norm': 23.832523927971703, 'learning_rate': 2.0952380952380955e-06, 'epoch': 1.62}\n",
      "{'loss': 0.0002, 'grad_norm': 0.01908084193494264, 'learning_rate': 2.0917107583774253e-06, 'epoch': 1.63}\n",
      "{'loss': 0.0008, 'grad_norm': 0.08678442952918536, 'learning_rate': 2.088183421516755e-06, 'epoch': 1.63}\n",
      "{'loss': 0.0052, 'grad_norm': 0.6165159410703603, 'learning_rate': 2.0846560846560847e-06, 'epoch': 1.63}\n",
      "{'loss': 0.0001, 'grad_norm': 0.007379620843125557, 'learning_rate': 2.0811287477954144e-06, 'epoch': 1.63}\n",
      "{'loss': 0.3059, 'grad_norm': 11.764506374238241, 'learning_rate': 2.0776014109347445e-06, 'epoch': 1.63}\n",
      "{'loss': 0.0013, 'grad_norm': 0.14664429518263342, 'learning_rate': 2.0740740740740742e-06, 'epoch': 1.63}\n",
      "{'loss': 0.0001, 'grad_norm': 0.0073792311350031, 'learning_rate': 2.070546737213404e-06, 'epoch': 1.63}\n",
      "{'loss': 0.0, 'grad_norm': 0.0008987088325577494, 'learning_rate': 2.0670194003527337e-06, 'epoch': 1.63}\n",
      "{'loss': 0.665, 'grad_norm': 16.69889299755376, 'learning_rate': 2.0634920634920634e-06, 'epoch': 1.63}\n",
      "{'loss': 0.1733, 'grad_norm': 6.796392875321633, 'learning_rate': 2.0599647266313935e-06, 'epoch': 1.63}\n",
      "{'loss': 0.0, 'grad_norm': 0.00042995027898858106, 'learning_rate': 2.0564373897707232e-06, 'epoch': 1.63}\n",
      "{'loss': 0.0001, 'grad_norm': 0.009967968921486098, 'learning_rate': 2.0529100529100534e-06, 'epoch': 1.63}\n",
      "{'loss': 0.0024, 'grad_norm': 0.2167105975608232, 'learning_rate': 2.049382716049383e-06, 'epoch': 1.63}\n",
      "{'loss': 0.0, 'grad_norm': 0.0005600418167447965, 'learning_rate': 2.0458553791887124e-06, 'epoch': 1.63}\n",
      "{'loss': 0.0, 'grad_norm': 3.783368146255796e-05, 'learning_rate': 2.0423280423280425e-06, 'epoch': 1.63}\n",
      "{'loss': 0.0, 'grad_norm': 0.001148384758618757, 'learning_rate': 2.0388007054673722e-06, 'epoch': 1.63}\n",
      "{'loss': 0.0, 'grad_norm': 0.0003074478875768616, 'learning_rate': 2.0352733686067024e-06, 'epoch': 1.64}\n",
      "{'loss': 0.0, 'grad_norm': 0.0034333270111508722, 'learning_rate': 2.031746031746032e-06, 'epoch': 1.64}\n",
      "{'loss': 0.3337, 'grad_norm': 18.53403867481692, 'learning_rate': 2.0282186948853618e-06, 'epoch': 1.64}\n",
      "{'loss': 0.0001, 'grad_norm': 0.013211046220187119, 'learning_rate': 2.0246913580246915e-06, 'epoch': 1.64}\n",
      "{'loss': 0.0002, 'grad_norm': 0.021442756385413885, 'learning_rate': 2.021164021164021e-06, 'epoch': 1.64}\n",
      "{'loss': 0.0002, 'grad_norm': 0.013726433710313415, 'learning_rate': 2.0176366843033513e-06, 'epoch': 1.64}\n",
      "{'loss': 0.0918, 'grad_norm': 11.996509680187648, 'learning_rate': 2.014109347442681e-06, 'epoch': 1.64}\n",
      "{'loss': 0.0, 'grad_norm': 0.0007290868216461185, 'learning_rate': 2.0105820105820108e-06, 'epoch': 1.64}\n",
      "{'loss': 0.0, 'grad_norm': 0.006172895095406455, 'learning_rate': 2.0070546737213405e-06, 'epoch': 1.64}\n",
      "{'loss': 0.0, 'grad_norm': 0.0002699311619880056, 'learning_rate': 2.00352733686067e-06, 'epoch': 1.64}\n",
      "{'loss': 0.0007, 'grad_norm': 0.1044453177105756, 'learning_rate': 2.0000000000000003e-06, 'epoch': 1.64}\n",
      "{'loss': 0.0699, 'grad_norm': 8.13140100548557, 'learning_rate': 1.99647266313933e-06, 'epoch': 1.64}\n",
      "{'loss': 0.0007, 'grad_norm': 0.10662194130262445, 'learning_rate': 1.9929453262786598e-06, 'epoch': 1.64}\n",
      "{'loss': 0.0, 'grad_norm': 0.002970856682092152, 'learning_rate': 1.9894179894179895e-06, 'epoch': 1.64}\n",
      "{'loss': 0.0, 'grad_norm': 0.0003918231273024884, 'learning_rate': 1.985890652557319e-06, 'epoch': 1.64}\n",
      "{'loss': 0.0, 'grad_norm': 0.0014722503051242401, 'learning_rate': 1.9823633156966493e-06, 'epoch': 1.65}\n",
      "{'loss': 0.0005, 'grad_norm': 0.07403734272819869, 'learning_rate': 1.978835978835979e-06, 'epoch': 1.65}\n",
      "{'loss': 0.0, 'grad_norm': 4.7382421725524024e-05, 'learning_rate': 1.9753086419753087e-06, 'epoch': 1.65}\n",
      "{'loss': 0.0007, 'grad_norm': 0.08199980110233016, 'learning_rate': 1.9717813051146385e-06, 'epoch': 1.65}\n",
      "{'loss': 0.0002, 'grad_norm': 0.018397426328135975, 'learning_rate': 1.968253968253968e-06, 'epoch': 1.65}\n",
      "{'loss': 0.0, 'grad_norm': 9.30529113366646e-05, 'learning_rate': 1.9647266313932983e-06, 'epoch': 1.65}\n",
      "{'loss': 0.0, 'grad_norm': 0.00522908141633704, 'learning_rate': 1.961199294532628e-06, 'epoch': 1.65}\n",
      "{'loss': 0.0, 'grad_norm': 0.002460127258189104, 'learning_rate': 1.9576719576719577e-06, 'epoch': 1.65}\n",
      "{'loss': 0.0, 'grad_norm': 0.0010497278982282328, 'learning_rate': 1.954144620811288e-06, 'epoch': 1.65}\n",
      "{'loss': 0.0014, 'grad_norm': 0.20434221198633024, 'learning_rate': 1.9506172839506176e-06, 'epoch': 1.65}\n",
      "{'loss': 0.0272, 'grad_norm': 3.2645556875753052, 'learning_rate': 1.9470899470899473e-06, 'epoch': 1.65}\n",
      "{'loss': 0.0, 'grad_norm': 0.0016874618927431126, 'learning_rate': 1.943562610229277e-06, 'epoch': 1.65}\n",
      "{'loss': 0.0224, 'grad_norm': 3.0190890483125243, 'learning_rate': 1.9400352733686067e-06, 'epoch': 1.65}\n",
      "{'loss': 0.0001, 'grad_norm': 0.006339172966810028, 'learning_rate': 1.936507936507937e-06, 'epoch': 1.65}\n",
      "{'loss': 0.0003, 'grad_norm': 0.03623767887717667, 'learning_rate': 1.9329805996472666e-06, 'epoch': 1.65}\n",
      "{'loss': 0.0, 'grad_norm': 0.003640193479438905, 'learning_rate': 1.9294532627865963e-06, 'epoch': 1.65}\n",
      "{'loss': 0.0001, 'grad_norm': 0.010748415396023227, 'learning_rate': 1.925925925925926e-06, 'epoch': 1.66}\n",
      "{'loss': 0.0, 'grad_norm': 0.0038823951406127656, 'learning_rate': 1.9223985890652557e-06, 'epoch': 1.66}\n",
      "{'loss': 0.0003, 'grad_norm': 0.03896848122629093, 'learning_rate': 1.918871252204586e-06, 'epoch': 1.66}\n",
      "{'loss': 0.0004, 'grad_norm': 0.0394095294460921, 'learning_rate': 1.9153439153439156e-06, 'epoch': 1.66}\n",
      "{'loss': 0.0, 'grad_norm': 0.0001157740825891082, 'learning_rate': 1.9118165784832453e-06, 'epoch': 1.66}\n",
      "{'loss': 0.0, 'grad_norm': 0.001218450598871926, 'learning_rate': 1.908289241622575e-06, 'epoch': 1.66}\n",
      "{'loss': 0.0002, 'grad_norm': 0.024509873738705137, 'learning_rate': 1.904761904761905e-06, 'epoch': 1.66}\n",
      "{'loss': 0.0008, 'grad_norm': 0.11997115554693212, 'learning_rate': 1.9012345679012348e-06, 'epoch': 1.66}\n",
      "{'loss': 0.0, 'grad_norm': 2.7444355469576363e-06, 'learning_rate': 1.8977072310405645e-06, 'epoch': 1.66}\n",
      "{'loss': 0.0, 'grad_norm': 2.012858175025531e-06, 'learning_rate': 1.8941798941798945e-06, 'epoch': 1.66}\n",
      "{'loss': 0.0001, 'grad_norm': 0.01452647637879181, 'learning_rate': 1.8906525573192242e-06, 'epoch': 1.66}\n",
      "{'loss': 0.1412, 'grad_norm': 12.025274997667672, 'learning_rate': 1.887125220458554e-06, 'epoch': 1.66}\n",
      "{'loss': 0.0026, 'grad_norm': 0.4461211341429617, 'learning_rate': 1.8835978835978838e-06, 'epoch': 1.66}\n",
      "{'loss': 0.0003, 'grad_norm': 0.028656805147149137, 'learning_rate': 1.8800705467372135e-06, 'epoch': 1.66}\n",
      "{'loss': 0.0, 'grad_norm': 5.130873581682497e-05, 'learning_rate': 1.8765432098765435e-06, 'epoch': 1.66}\n",
      "{'loss': 0.0, 'grad_norm': 2.513946460072352e-05, 'learning_rate': 1.8730158730158732e-06, 'epoch': 1.66}\n",
      "{'loss': 0.0, 'grad_norm': 0.0002798235764569295, 'learning_rate': 1.8694885361552029e-06, 'epoch': 1.67}\n",
      "{'loss': 0.0, 'grad_norm': 3.992567168615808e-06, 'learning_rate': 1.8659611992945328e-06, 'epoch': 1.67}\n",
      "{'loss': 0.0034, 'grad_norm': 0.42332583429547993, 'learning_rate': 1.8624338624338625e-06, 'epoch': 1.67}\n",
      "{'loss': 0.0005, 'grad_norm': 0.04980190587223252, 'learning_rate': 1.8589065255731924e-06, 'epoch': 1.67}\n",
      "{'loss': 0.0001, 'grad_norm': 0.010979972762781457, 'learning_rate': 1.8553791887125222e-06, 'epoch': 1.67}\n",
      "{'loss': 0.1357, 'grad_norm': 9.36044116944916, 'learning_rate': 1.8518518518518519e-06, 'epoch': 1.67}\n",
      "{'loss': 0.0, 'grad_norm': 0.0027856650477618376, 'learning_rate': 1.8483245149911818e-06, 'epoch': 1.67}\n",
      "{'loss': 0.0, 'grad_norm': 0.0028420145872194136, 'learning_rate': 1.8447971781305115e-06, 'epoch': 1.67}\n",
      "{'loss': 0.0, 'grad_norm': 0.0002987178693962375, 'learning_rate': 1.8412698412698416e-06, 'epoch': 1.67}\n",
      "{'loss': 0.0002, 'grad_norm': 0.0286473575009549, 'learning_rate': 1.8377425044091711e-06, 'epoch': 1.67}\n",
      "{'loss': 0.0, 'grad_norm': 0.00010965769686019278, 'learning_rate': 1.8342151675485009e-06, 'epoch': 1.67}\n",
      "{'loss': 0.0, 'grad_norm': 0.00038733261053290656, 'learning_rate': 1.830687830687831e-06, 'epoch': 1.67}\n",
      "{'loss': 0.0, 'grad_norm': 8.29080174379057e-05, 'learning_rate': 1.8271604938271605e-06, 'epoch': 1.67}\n",
      "{'loss': 0.0, 'grad_norm': 3.687232983708937e-06, 'learning_rate': 1.8236331569664906e-06, 'epoch': 1.67}\n",
      "{'loss': 0.0001, 'grad_norm': 0.013553490008685014, 'learning_rate': 1.8201058201058203e-06, 'epoch': 1.67}\n",
      "{'loss': 0.0, 'grad_norm': 8.748768054589306e-05, 'learning_rate': 1.81657848324515e-06, 'epoch': 1.67}\n",
      "{'loss': 0.0, 'grad_norm': 0.0006136490827846181, 'learning_rate': 1.81305114638448e-06, 'epoch': 1.68}\n",
      "{'loss': 0.0, 'grad_norm': 0.0024061880544405732, 'learning_rate': 1.8095238095238097e-06, 'epoch': 1.68}\n",
      "{'loss': 0.0, 'grad_norm': 0.0027076555527271373, 'learning_rate': 1.8059964726631396e-06, 'epoch': 1.68}\n",
      "{'loss': 0.0001, 'grad_norm': 0.006878144592544199, 'learning_rate': 1.8024691358024693e-06, 'epoch': 1.68}\n",
      "{'loss': 0.0, 'grad_norm': 7.3013929343864665e-06, 'learning_rate': 1.798941798941799e-06, 'epoch': 1.68}\n",
      "{'loss': 0.0, 'grad_norm': 6.651058475494532e-05, 'learning_rate': 1.795414462081129e-06, 'epoch': 1.68}\n",
      "{'loss': 0.0, 'grad_norm': 1.629305189456473e-05, 'learning_rate': 1.7918871252204587e-06, 'epoch': 1.68}\n",
      "{'loss': 0.2844, 'grad_norm': 22.883751367731566, 'learning_rate': 1.7883597883597886e-06, 'epoch': 1.68}\n",
      "{'loss': 0.0, 'grad_norm': 1.0898030992083968e-05, 'learning_rate': 1.7848324514991183e-06, 'epoch': 1.68}\n",
      "{'loss': 0.0029, 'grad_norm': 0.45395672857249786, 'learning_rate': 1.781305114638448e-06, 'epoch': 1.68}\n",
      "{'loss': 0.0, 'grad_norm': 0.0023853452724112985, 'learning_rate': 1.777777777777778e-06, 'epoch': 1.68}\n",
      "{'loss': 0.0, 'grad_norm': 8.969104823029639e-07, 'learning_rate': 1.7742504409171077e-06, 'epoch': 1.68}\n",
      "{'loss': 0.0, 'grad_norm': 0.0008562855628749842, 'learning_rate': 1.7707231040564376e-06, 'epoch': 1.68}\n",
      "{'loss': 0.0, 'grad_norm': 0.0003675753638260854, 'learning_rate': 1.7671957671957673e-06, 'epoch': 1.68}\n",
      "{'loss': 0.0, 'grad_norm': 0.0007468930725981942, 'learning_rate': 1.763668430335097e-06, 'epoch': 1.68}\n",
      "{'loss': 0.0032, 'grad_norm': 0.2535838207338805, 'learning_rate': 1.760141093474427e-06, 'epoch': 1.69}\n",
      "{'loss': 0.0, 'grad_norm': 0.0024464973251721064, 'learning_rate': 1.7566137566137567e-06, 'epoch': 1.69}\n",
      "{'loss': 0.0002, 'grad_norm': 0.019742021398964537, 'learning_rate': 1.7530864197530868e-06, 'epoch': 1.69}\n",
      "{'loss': 0.0002, 'grad_norm': 0.03627827400316081, 'learning_rate': 1.7495590828924163e-06, 'epoch': 1.69}\n",
      "{'loss': 0.0, 'grad_norm': 0.0008937580837258351, 'learning_rate': 1.746031746031746e-06, 'epoch': 1.69}\n",
      "{'loss': 0.0, 'grad_norm': 0.0004179667755762465, 'learning_rate': 1.7425044091710761e-06, 'epoch': 1.69}\n",
      "{'loss': 0.0005, 'grad_norm': 0.058672238471331406, 'learning_rate': 1.7389770723104056e-06, 'epoch': 1.69}\n",
      "{'loss': 0.0, 'grad_norm': 0.0002482083839169945, 'learning_rate': 1.7354497354497358e-06, 'epoch': 1.69}\n",
      "{'loss': 0.0002, 'grad_norm': 0.024780982810705062, 'learning_rate': 1.7319223985890655e-06, 'epoch': 1.69}\n",
      "{'loss': 0.0, 'grad_norm': 0.00016520204464278378, 'learning_rate': 1.7283950617283952e-06, 'epoch': 1.69}\n",
      "{'loss': 0.0, 'grad_norm': 0.00044707814642590965, 'learning_rate': 1.7248677248677251e-06, 'epoch': 1.69}\n",
      "{'loss': 0.0, 'grad_norm': 1.4225758409631986e-05, 'learning_rate': 1.7213403880070548e-06, 'epoch': 1.69}\n",
      "{'loss': 0.0981, 'grad_norm': 5.299050753535939, 'learning_rate': 1.7178130511463848e-06, 'epoch': 1.69}\n",
      "{'loss': 0.0025, 'grad_norm': 0.3606017776628453, 'learning_rate': 1.7142857142857145e-06, 'epoch': 1.69}\n",
      "{'loss': 0.0001, 'grad_norm': 0.011367270438229259, 'learning_rate': 1.7107583774250442e-06, 'epoch': 1.69}\n",
      "{'loss': 0.0045, 'grad_norm': 0.5480397219201286, 'learning_rate': 1.7072310405643741e-06, 'epoch': 1.69}\n",
      "{'loss': 0.0, 'grad_norm': 1.544580660839443e-06, 'learning_rate': 1.7037037037037038e-06, 'epoch': 1.7}\n",
      "{'loss': 0.0008, 'grad_norm': 0.09050890643232933, 'learning_rate': 1.7001763668430338e-06, 'epoch': 1.7}\n",
      "{'loss': 0.0, 'grad_norm': 3.178818654379665e-05, 'learning_rate': 1.6966490299823635e-06, 'epoch': 1.7}\n",
      "{'loss': 0.0, 'grad_norm': 8.89986237202205e-06, 'learning_rate': 1.6931216931216932e-06, 'epoch': 1.7}\n",
      "{'loss': 0.0, 'grad_norm': 6.8871482596971246e-06, 'learning_rate': 1.689594356261023e-06, 'epoch': 1.7}\n",
      "{'loss': 0.0, 'grad_norm': 0.0024422362898427466, 'learning_rate': 1.6860670194003528e-06, 'epoch': 1.7}\n",
      "{'loss': 0.001, 'grad_norm': 0.12624494662888394, 'learning_rate': 1.6825396825396827e-06, 'epoch': 1.7}\n",
      "{'loss': 0.0, 'grad_norm': 0.0035241809506783846, 'learning_rate': 1.6790123456790125e-06, 'epoch': 1.7}\n",
      "{'loss': 0.0, 'grad_norm': 0.004633289825665899, 'learning_rate': 1.6754850088183422e-06, 'epoch': 1.7}\n",
      "{'loss': 0.0123, 'grad_norm': 1.7640209507336004, 'learning_rate': 1.671957671957672e-06, 'epoch': 1.7}\n",
      "{'loss': 0.5522, 'grad_norm': 17.333999229747157, 'learning_rate': 1.6684303350970018e-06, 'epoch': 1.7}\n",
      "{'loss': 0.0325, 'grad_norm': 2.8296969771160665, 'learning_rate': 1.6649029982363317e-06, 'epoch': 1.7}\n",
      "{'loss': 0.0, 'grad_norm': 0.00064173043437846, 'learning_rate': 1.6613756613756614e-06, 'epoch': 1.7}\n",
      "{'loss': 0.0, 'grad_norm': 0.0002211613778558736, 'learning_rate': 1.6578483245149912e-06, 'epoch': 1.7}\n",
      "{'loss': 0.0018, 'grad_norm': 0.23773520079339874, 'learning_rate': 1.6543209876543213e-06, 'epoch': 1.7}\n",
      "{'loss': 0.0, 'grad_norm': 0.0003616905517607784, 'learning_rate': 1.6507936507936508e-06, 'epoch': 1.7}\n",
      "{'loss': 0.0, 'grad_norm': 0.00048277282356808956, 'learning_rate': 1.647266313932981e-06, 'epoch': 1.71}\n",
      "{'loss': 0.0001, 'grad_norm': 0.010048097972275034, 'learning_rate': 1.6437389770723106e-06, 'epoch': 1.71}\n",
      "{'loss': 0.0, 'grad_norm': 1.2268966527450298e-06, 'learning_rate': 1.6402116402116404e-06, 'epoch': 1.71}\n",
      "{'loss': 0.0, 'grad_norm': 0.004367792970605526, 'learning_rate': 1.6366843033509703e-06, 'epoch': 1.71}\n",
      "{'loss': 0.0, 'grad_norm': 7.590987029502902e-05, 'learning_rate': 1.6331569664903e-06, 'epoch': 1.71}\n",
      "{'loss': 0.0, 'grad_norm': 2.7200482051652248e-05, 'learning_rate': 1.62962962962963e-06, 'epoch': 1.71}\n",
      "{'loss': 0.0, 'grad_norm': 1.8573238686293525e-05, 'learning_rate': 1.6261022927689596e-06, 'epoch': 1.71}\n",
      "{'loss': 0.0, 'grad_norm': 1.7528995553588954e-05, 'learning_rate': 1.6225749559082893e-06, 'epoch': 1.71}\n",
      "{'loss': 0.0005, 'grad_norm': 0.03977620966194727, 'learning_rate': 1.6190476190476193e-06, 'epoch': 1.71}\n",
      "{'loss': 0.0101, 'grad_norm': 1.9554110766572816, 'learning_rate': 1.615520282186949e-06, 'epoch': 1.71}\n",
      "{'loss': 0.0, 'grad_norm': 7.037358089467794e-06, 'learning_rate': 1.611992945326279e-06, 'epoch': 1.71}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 86%|████████▌ | 2696/3150 [11:12<01:56,  3.88it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.0, 'grad_norm': 0.002211572384720586, 'learning_rate': 1.6084656084656086e-06, 'epoch': 1.71}\n",
      "{'loss': 0.0086, 'grad_norm': 0.68364602025328, 'learning_rate': 1.6049382716049383e-06, 'epoch': 1.71}\n",
      "{'loss': 0.0003, 'grad_norm': 0.026921721577691494, 'learning_rate': 1.6014109347442683e-06, 'epoch': 1.71}\n",
      "{'loss': 0.0, 'grad_norm': 0.0001044867510325982, 'learning_rate': 1.597883597883598e-06, 'epoch': 1.71}\n",
      "{'loss': 0.0, 'grad_norm': 0.00025342561809199815, 'learning_rate': 1.5943562610229279e-06, 'epoch': 1.71}\n",
      "{'loss': 0.0, 'grad_norm': 2.4368396991437897e-05, 'learning_rate': 1.5908289241622576e-06, 'epoch': 1.72}\n",
      "{'loss': 0.0, 'grad_norm': 5.531408833197814e-06, 'learning_rate': 1.5873015873015873e-06, 'epoch': 1.72}\n",
      "{'loss': 0.0, 'grad_norm': 9.209012136005157e-05, 'learning_rate': 1.5837742504409172e-06, 'epoch': 1.72}\n",
      "{'loss': 0.0001, 'grad_norm': 0.015541857089792681, 'learning_rate': 1.580246913580247e-06, 'epoch': 1.72}\n",
      "{'loss': 0.0, 'grad_norm': 0.0006011956716473188, 'learning_rate': 1.5767195767195769e-06, 'epoch': 1.72}\n",
      "{'loss': 0.0, 'grad_norm': 0.0008768041872332969, 'learning_rate': 1.5731922398589066e-06, 'epoch': 1.72}\n",
      "{'loss': 0.0, 'grad_norm': 0.00020474367265938462, 'learning_rate': 1.5696649029982363e-06, 'epoch': 1.72}\n",
      "{'loss': 0.0018, 'grad_norm': 0.22843636662506034, 'learning_rate': 1.5661375661375664e-06, 'epoch': 1.72}\n",
      "{'loss': 0.0, 'grad_norm': 0.0007140275423861552, 'learning_rate': 1.562610229276896e-06, 'epoch': 1.72}\n",
      "{'loss': 0.0, 'grad_norm': 5.893887256758022e-06, 'learning_rate': 1.559082892416226e-06, 'epoch': 1.72}\n",
      "{'loss': 0.0, 'grad_norm': 0.0002448551262469679, 'learning_rate': 1.5555555555555558e-06, 'epoch': 1.72}\n",
      "{'loss': 0.0001, 'grad_norm': 0.007650143542090581, 'learning_rate': 1.5520282186948855e-06, 'epoch': 1.72}\n",
      "{'loss': 0.0, 'grad_norm': 0.000680595285974768, 'learning_rate': 1.5485008818342154e-06, 'epoch': 1.72}\n",
      "{'loss': 0.0079, 'grad_norm': 0.8576919373662464, 'learning_rate': 1.5449735449735451e-06, 'epoch': 1.72}\n",
      "{'loss': 0.0, 'grad_norm': 0.003081882328994242, 'learning_rate': 1.541446208112875e-06, 'epoch': 1.72}\n",
      "{'loss': 0.0009, 'grad_norm': 0.22326025268137406, 'learning_rate': 1.5379188712522048e-06, 'epoch': 1.73}\n",
      "{'loss': 0.0006, 'grad_norm': 0.08452600933107397, 'learning_rate': 1.5343915343915345e-06, 'epoch': 1.73}\n",
      "{'loss': 0.0, 'grad_norm': 5.8263281235834025e-06, 'learning_rate': 1.5308641975308644e-06, 'epoch': 1.73}\n",
      "{'loss': 0.0, 'grad_norm': 2.8641097824738384e-05, 'learning_rate': 1.5273368606701941e-06, 'epoch': 1.73}\n",
      "{'loss': 0.0, 'grad_norm': 0.00023953284712563937, 'learning_rate': 1.523809523809524e-06, 'epoch': 1.73}\n",
      "{'loss': 0.0009, 'grad_norm': 0.10318961178474603, 'learning_rate': 1.5202821869488538e-06, 'epoch': 1.73}\n",
      "{'loss': 0.0, 'grad_norm': 0.0047201882685617294, 'learning_rate': 1.5167548500881835e-06, 'epoch': 1.73}\n",
      "{'loss': 0.2201, 'grad_norm': 13.46637239292025, 'learning_rate': 1.5132275132275134e-06, 'epoch': 1.73}\n",
      "{'loss': 0.0002, 'grad_norm': 0.031228290780723086, 'learning_rate': 1.5097001763668431e-06, 'epoch': 1.73}\n",
      "{'loss': 0.0, 'grad_norm': 1.2915673334161088e-05, 'learning_rate': 1.506172839506173e-06, 'epoch': 1.73}\n",
      "{'loss': 0.0, 'grad_norm': 0.00026499186054424075, 'learning_rate': 1.5026455026455028e-06, 'epoch': 1.73}\n",
      "{'loss': 0.0003, 'grad_norm': 0.02760296173622346, 'learning_rate': 1.4991181657848325e-06, 'epoch': 1.73}\n",
      "{'loss': 0.0, 'grad_norm': 0.0001639119874328509, 'learning_rate': 1.4955908289241624e-06, 'epoch': 1.73}\n",
      "{'loss': 0.0, 'grad_norm': 0.0002642466236169019, 'learning_rate': 1.492063492063492e-06, 'epoch': 1.73}\n",
      "{'loss': 0.0001, 'grad_norm': 0.01049754403540176, 'learning_rate': 1.4885361552028218e-06, 'epoch': 1.73}\n",
      "{'loss': 0.0, 'grad_norm': 7.549260044073239e-05, 'learning_rate': 1.4850088183421517e-06, 'epoch': 1.73}\n",
      "{'loss': 0.0004, 'grad_norm': 0.050830771068646474, 'learning_rate': 1.4814814814814815e-06, 'epoch': 1.74}\n",
      "{'loss': 0.0204, 'grad_norm': 2.0886125306801664, 'learning_rate': 1.4779541446208116e-06, 'epoch': 1.74}\n",
      "{'loss': 0.0, 'grad_norm': 7.874933377297752e-05, 'learning_rate': 1.474426807760141e-06, 'epoch': 1.74}\n",
      "{'loss': 0.0, 'grad_norm': 0.00033736621741511957, 'learning_rate': 1.4708994708994708e-06, 'epoch': 1.74}\n",
      "{'loss': 0.0074, 'grad_norm': 0.8144405007241585, 'learning_rate': 1.467372134038801e-06, 'epoch': 1.74}\n",
      "{'loss': 0.0, 'grad_norm': 0.0030701715044873356, 'learning_rate': 1.4638447971781307e-06, 'epoch': 1.74}\n",
      "{'loss': 0.1327, 'grad_norm': 15.034381881201657, 'learning_rate': 1.4603174603174606e-06, 'epoch': 1.74}\n",
      "{'loss': 0.0, 'grad_norm': 6.829819750438748e-05, 'learning_rate': 1.4567901234567903e-06, 'epoch': 1.74}\n",
      "{'loss': 0.0, 'grad_norm': 0.006652420254309143, 'learning_rate': 1.45326278659612e-06, 'epoch': 1.74}\n",
      "{'loss': 0.0, 'grad_norm': 0.004331160809346781, 'learning_rate': 1.44973544973545e-06, 'epoch': 1.74}\n",
      "{'loss': 0.0, 'grad_norm': 5.683233725647659e-05, 'learning_rate': 1.4462081128747796e-06, 'epoch': 1.74}\n",
      "{'loss': 0.2377, 'grad_norm': 11.043771871772874, 'learning_rate': 1.4426807760141096e-06, 'epoch': 1.74}\n",
      "{'loss': 0.0, 'grad_norm': 0.0009107596275067593, 'learning_rate': 1.4391534391534393e-06, 'epoch': 1.74}\n",
      "{'loss': 0.0, 'grad_norm': 5.805927448242829e-05, 'learning_rate': 1.435626102292769e-06, 'epoch': 1.74}\n",
      "{'loss': 0.0, 'grad_norm': 0.00016838268368588136, 'learning_rate': 1.432098765432099e-06, 'epoch': 1.74}\n",
      "{'loss': 0.0, 'grad_norm': 0.00018482000034348296, 'learning_rate': 1.4285714285714286e-06, 'epoch': 1.74}\n",
      "{'loss': 0.0005, 'grad_norm': 0.04293543163239497, 'learning_rate': 1.4250440917107586e-06, 'epoch': 1.75}\n",
      "{'loss': 0.0, 'grad_norm': 0.00015671997745973104, 'learning_rate': 1.4215167548500883e-06, 'epoch': 1.75}\n",
      "{'loss': 0.0, 'grad_norm': 0.002146207764222848, 'learning_rate': 1.417989417989418e-06, 'epoch': 1.75}\n",
      "{'loss': 1.252, 'grad_norm': 27.72300493167013, 'learning_rate': 1.414462081128748e-06, 'epoch': 1.75}\n",
      "{'loss': 0.0, 'grad_norm': 0.002552250388097407, 'learning_rate': 1.4109347442680776e-06, 'epoch': 1.75}\n",
      "{'loss': 0.0, 'grad_norm': 0.001347098518899094, 'learning_rate': 1.4074074074074075e-06, 'epoch': 1.75}\n",
      "{'loss': 0.0044, 'grad_norm': 0.6019769949763011, 'learning_rate': 1.4038800705467373e-06, 'epoch': 1.75}\n",
      "{'loss': 0.0, 'grad_norm': 0.0017302346256344854, 'learning_rate': 1.400352733686067e-06, 'epoch': 1.75}\n",
      "{'loss': 0.0, 'grad_norm': 0.0035420531008771993, 'learning_rate': 1.3968253968253969e-06, 'epoch': 1.75}\n",
      "{'loss': 0.0027, 'grad_norm': 0.3719417720441004, 'learning_rate': 1.3932980599647266e-06, 'epoch': 1.75}\n",
      "{'loss': 0.3171, 'grad_norm': 12.065220659422142, 'learning_rate': 1.3897707231040567e-06, 'epoch': 1.75}\n",
      "{'loss': 0.0002, 'grad_norm': 0.024788508667591407, 'learning_rate': 1.3862433862433862e-06, 'epoch': 1.75}\n",
      "{'loss': 0.0, 'grad_norm': 0.00019248097953165265, 'learning_rate': 1.382716049382716e-06, 'epoch': 1.75}\n",
      "{'loss': 0.0, 'grad_norm': 0.004039566505274838, 'learning_rate': 1.379188712522046e-06, 'epoch': 1.75}\n",
      "{'loss': 0.0, 'grad_norm': 1.1181118539189842e-06, 'learning_rate': 1.3756613756613758e-06, 'epoch': 1.75}\n",
      "{'loss': 0.0, 'grad_norm': 4.025972578790417e-07, 'learning_rate': 1.3721340388007057e-06, 'epoch': 1.75}\n",
      "{'loss': 0.0, 'grad_norm': 0.00317073921589386, 'learning_rate': 1.3686067019400354e-06, 'epoch': 1.76}\n",
      "{'loss': 0.0, 'grad_norm': 6.20577599257636e-06, 'learning_rate': 1.3650793650793652e-06, 'epoch': 1.76}\n",
      "{'loss': 0.0006, 'grad_norm': 0.06733579999704083, 'learning_rate': 1.361552028218695e-06, 'epoch': 1.76}\n",
      "{'loss': 0.0, 'grad_norm': 2.07316175760067e-07, 'learning_rate': 1.3580246913580248e-06, 'epoch': 1.76}\n",
      "{'loss': 0.1733, 'grad_norm': 7.135452366946123, 'learning_rate': 1.3544973544973547e-06, 'epoch': 1.76}\n",
      "{'loss': 0.0015, 'grad_norm': 0.2206701321542835, 'learning_rate': 1.3509700176366844e-06, 'epoch': 1.76}\n",
      "{'loss': 0.0, 'grad_norm': 0.0006435399966684016, 'learning_rate': 1.3474426807760141e-06, 'epoch': 1.76}\n",
      "{'loss': 0.0, 'grad_norm': 0.0009882830588329272, 'learning_rate': 1.343915343915344e-06, 'epoch': 1.76}\n",
      "{'loss': 0.0, 'grad_norm': 2.975506978323271e-06, 'learning_rate': 1.3403880070546738e-06, 'epoch': 1.76}\n",
      "{'loss': 0.0209, 'grad_norm': 2.3429470467015383, 'learning_rate': 1.3368606701940037e-06, 'epoch': 1.76}\n",
      "{'loss': 0.0, 'grad_norm': 0.003592179363099259, 'learning_rate': 1.3333333333333334e-06, 'epoch': 1.76}\n",
      "{'loss': 0.0, 'grad_norm': 0.0007723002321663679, 'learning_rate': 1.3298059964726631e-06, 'epoch': 1.76}\n",
      "{'loss': 0.0002, 'grad_norm': 0.01848894710926474, 'learning_rate': 1.326278659611993e-06, 'epoch': 1.76}\n",
      "{'loss': 0.0, 'grad_norm': 0.0006335717036339369, 'learning_rate': 1.3227513227513228e-06, 'epoch': 1.76}\n",
      "{'loss': 0.0, 'grad_norm': 0.0037686510993359106, 'learning_rate': 1.3192239858906527e-06, 'epoch': 1.76}\n",
      "{'loss': 0.0003, 'grad_norm': 0.04624992327506557, 'learning_rate': 1.3156966490299824e-06, 'epoch': 1.77}\n",
      "{'loss': 0.0, 'grad_norm': 8.392777144638153e-05, 'learning_rate': 1.3121693121693121e-06, 'epoch': 1.77}\n",
      "{'loss': 0.0, 'grad_norm': 0.00044583075808159953, 'learning_rate': 1.308641975308642e-06, 'epoch': 1.77}\n",
      "{'loss': 0.0, 'grad_norm': 2.192704413988218e-06, 'learning_rate': 1.3051146384479718e-06, 'epoch': 1.77}\n",
      "{'loss': 0.0, 'grad_norm': 6.642638493784298e-05, 'learning_rate': 1.3015873015873019e-06, 'epoch': 1.77}\n",
      "{'loss': 0.0, 'grad_norm': 7.4113463796958506e-06, 'learning_rate': 1.2980599647266314e-06, 'epoch': 1.77}\n",
      "{'loss': 0.0011, 'grad_norm': 0.1432114669671816, 'learning_rate': 1.294532627865961e-06, 'epoch': 1.77}\n",
      "{'loss': 0.0, 'grad_norm': 0.0004176969497779919, 'learning_rate': 1.2910052910052912e-06, 'epoch': 1.77}\n",
      "{'loss': 0.0, 'grad_norm': 3.917324970278112e-05, 'learning_rate': 1.287477954144621e-06, 'epoch': 1.77}\n",
      "{'loss': 0.0, 'grad_norm': 3.3993425791463455e-06, 'learning_rate': 1.2839506172839509e-06, 'epoch': 1.77}\n",
      "{'loss': 0.0, 'grad_norm': 0.0042048165732902956, 'learning_rate': 1.2804232804232806e-06, 'epoch': 1.77}\n",
      "{'loss': 0.0, 'grad_norm': 2.8501663153868015e-05, 'learning_rate': 1.2768959435626103e-06, 'epoch': 1.77}\n",
      "{'loss': 0.0, 'grad_norm': 4.29597732990206e-06, 'learning_rate': 1.2733686067019402e-06, 'epoch': 1.77}\n",
      "{'loss': 0.0, 'grad_norm': 0.002373042482372769, 'learning_rate': 1.26984126984127e-06, 'epoch': 1.77}\n",
      "{'loss': 0.0, 'grad_norm': 0.0021873084390146686, 'learning_rate': 1.2663139329805999e-06, 'epoch': 1.77}\n",
      "{'loss': 0.2588, 'grad_norm': 6.179318963212782, 'learning_rate': 1.2627865961199296e-06, 'epoch': 1.77}\n",
      "{'loss': 0.0, 'grad_norm': 1.354229571593673e-06, 'learning_rate': 1.2592592592592593e-06, 'epoch': 1.78}\n",
      "{'loss': 0.0, 'grad_norm': 0.0009644855197677818, 'learning_rate': 1.2557319223985892e-06, 'epoch': 1.78}\n",
      "{'loss': 0.0, 'grad_norm': 0.003116933468256328, 'learning_rate': 1.252204585537919e-06, 'epoch': 1.78}\n",
      "{'loss': 0.0, 'grad_norm': 4.6134710463368e-06, 'learning_rate': 1.2486772486772486e-06, 'epoch': 1.78}\n",
      "{'loss': 0.0, 'grad_norm': 0.0003753856235895795, 'learning_rate': 1.2451499118165786e-06, 'epoch': 1.78}\n",
      "{'loss': 0.0002, 'grad_norm': 0.02089706412486388, 'learning_rate': 1.2416225749559085e-06, 'epoch': 1.78}\n",
      "{'loss': 0.0, 'grad_norm': 0.00021344479885947745, 'learning_rate': 1.2380952380952382e-06, 'epoch': 1.78}\n",
      "{'loss': 0.0, 'grad_norm': 5.7142984525768496e-05, 'learning_rate': 1.234567901234568e-06, 'epoch': 1.78}\n",
      "{'loss': 0.0025, 'grad_norm': 0.29267618193931505, 'learning_rate': 1.2310405643738978e-06, 'epoch': 1.78}\n",
      "{'loss': 0.0019, 'grad_norm': 0.26363492636172975, 'learning_rate': 1.2275132275132276e-06, 'epoch': 1.78}\n",
      "{'loss': 0.0, 'grad_norm': 0.0003642296411761232, 'learning_rate': 1.2239858906525575e-06, 'epoch': 1.78}\n",
      "{'loss': 0.0, 'grad_norm': 0.0006580077429898279, 'learning_rate': 1.2204585537918872e-06, 'epoch': 1.78}\n",
      "{'loss': 0.0, 'grad_norm': 0.00016092692250581673, 'learning_rate': 1.216931216931217e-06, 'epoch': 1.78}\n",
      "{'loss': 0.0, 'grad_norm': 0.0012291007534637646, 'learning_rate': 1.2134038800705468e-06, 'epoch': 1.78}\n",
      "{'loss': 0.0003, 'grad_norm': 0.028826454453085168, 'learning_rate': 1.2098765432098765e-06, 'epoch': 1.78}\n",
      "{'loss': 0.0003, 'grad_norm': 0.03249194411298744, 'learning_rate': 1.2063492063492065e-06, 'epoch': 1.78}\n",
      "{'loss': 0.0001, 'grad_norm': 0.017417518387468998, 'learning_rate': 1.2028218694885364e-06, 'epoch': 1.79}\n",
      "{'loss': 0.0, 'grad_norm': 0.00013580253515009232, 'learning_rate': 1.1992945326278659e-06, 'epoch': 1.79}\n",
      "{'loss': 0.0, 'grad_norm': 2.0909304794310134e-05, 'learning_rate': 1.1957671957671958e-06, 'epoch': 1.79}\n",
      "{'loss': 0.0, 'grad_norm': 0.005110268963462991, 'learning_rate': 1.1922398589065257e-06, 'epoch': 1.79}\n",
      "{'loss': 0.0001, 'grad_norm': 0.00801157039981475, 'learning_rate': 1.1887125220458555e-06, 'epoch': 1.79}\n",
      "{'loss': 0.0, 'grad_norm': 0.004335488193957374, 'learning_rate': 1.1851851851851854e-06, 'epoch': 1.79}\n",
      "{'loss': 0.0533, 'grad_norm': 5.022127588800236, 'learning_rate': 1.181657848324515e-06, 'epoch': 1.79}\n",
      "{'loss': 0.0024, 'grad_norm': 0.26110115059044264, 'learning_rate': 1.1781305114638448e-06, 'epoch': 1.79}\n",
      "{'loss': 0.0, 'grad_norm': 8.712455337354968e-06, 'learning_rate': 1.1746031746031747e-06, 'epoch': 1.79}\n",
      "{'loss': 0.0003, 'grad_norm': 0.03798988805917695, 'learning_rate': 1.1710758377425044e-06, 'epoch': 1.79}\n",
      "{'loss': 0.0, 'grad_norm': 7.93437931071683e-05, 'learning_rate': 1.1675485008818344e-06, 'epoch': 1.79}\n",
      "{'loss': 0.0002, 'grad_norm': 0.03274210463058044, 'learning_rate': 1.164021164021164e-06, 'epoch': 1.79}\n",
      "{'loss': 0.0, 'grad_norm': 0.00152468433409751, 'learning_rate': 1.1604938271604938e-06, 'epoch': 1.79}\n",
      "{'loss': 0.0, 'grad_norm': 1.2508801952944504e-07, 'learning_rate': 1.1569664902998237e-06, 'epoch': 1.79}\n",
      "{'loss': 0.0, 'grad_norm': 1.5108838751115796e-05, 'learning_rate': 1.1534391534391536e-06, 'epoch': 1.79}\n",
      "{'loss': 0.0278, 'grad_norm': 2.900336535427879, 'learning_rate': 1.1499118165784833e-06, 'epoch': 1.79}\n",
      "{'loss': 0.0004, 'grad_norm': 0.05175193717286274, 'learning_rate': 1.146384479717813e-06, 'epoch': 1.8}\n",
      "{'loss': 0.0, 'grad_norm': 4.0214890961743556e-05, 'learning_rate': 1.142857142857143e-06, 'epoch': 1.8}\n",
      "{'loss': 0.0, 'grad_norm': 0.0010062250997397715, 'learning_rate': 1.1393298059964727e-06, 'epoch': 1.8}\n",
      "{'loss': 0.0, 'grad_norm': 0.00019611686298723794, 'learning_rate': 1.1358024691358026e-06, 'epoch': 1.8}\n",
      "{'loss': 0.0014, 'grad_norm': 0.19292418316629523, 'learning_rate': 1.1322751322751323e-06, 'epoch': 1.8}\n",
      "{'loss': 0.0007, 'grad_norm': 0.09123469233306612, 'learning_rate': 1.128747795414462e-06, 'epoch': 1.8}\n",
      "{'loss': 0.0, 'grad_norm': 0.002106139905969461, 'learning_rate': 1.125220458553792e-06, 'epoch': 1.8}\n",
      "{'loss': 0.0, 'grad_norm': 0.0002760885730514098, 'learning_rate': 1.1216931216931217e-06, 'epoch': 1.8}\n",
      "{'loss': 0.0, 'grad_norm': 0.00014326180032953322, 'learning_rate': 1.1181657848324516e-06, 'epoch': 1.8}\n",
      "{'loss': 0.0, 'grad_norm': 0.004750953523686838, 'learning_rate': 1.1146384479717815e-06, 'epoch': 1.8}\n",
      "{'loss': 0.0017, 'grad_norm': 0.18708496053925416, 'learning_rate': 1.111111111111111e-06, 'epoch': 1.8}\n",
      "{'loss': 0.0, 'grad_norm': 1.8095478393047578e-06, 'learning_rate': 1.107583774250441e-06, 'epoch': 1.8}\n",
      "{'loss': 0.0001, 'grad_norm': 0.007982430176973284, 'learning_rate': 1.1040564373897709e-06, 'epoch': 1.8}\n",
      "{'loss': 0.0029, 'grad_norm': 0.3212281491125167, 'learning_rate': 1.1005291005291006e-06, 'epoch': 1.8}\n",
      "{'loss': 0.0, 'grad_norm': 5.996702962976852e-06, 'learning_rate': 1.0970017636684305e-06, 'epoch': 1.8}\n",
      "{'loss': 0.0, 'grad_norm': 0.0003011159993330132, 'learning_rate': 1.0934744268077602e-06, 'epoch': 1.81}\n",
      "{'loss': 0.0185, 'grad_norm': 2.2248727912113884, 'learning_rate': 1.08994708994709e-06, 'epoch': 1.81}\n",
      "{'loss': 0.0, 'grad_norm': 2.8048640603599284e-05, 'learning_rate': 1.0864197530864199e-06, 'epoch': 1.81}\n",
      "{'loss': 0.0, 'grad_norm': 0.0009734906487685746, 'learning_rate': 1.0828924162257496e-06, 'epoch': 1.81}\n",
      "{'loss': 0.0014, 'grad_norm': 0.139163776819078, 'learning_rate': 1.0793650793650795e-06, 'epoch': 1.81}\n",
      "{'loss': 0.0005, 'grad_norm': 0.06110805735414777, 'learning_rate': 1.0758377425044092e-06, 'epoch': 1.81}\n",
      "{'loss': 0.0, 'grad_norm': 2.9046019118464493e-06, 'learning_rate': 1.072310405643739e-06, 'epoch': 1.81}\n",
      "{'loss': 0.0198, 'grad_norm': 2.7254712563487713, 'learning_rate': 1.0687830687830689e-06, 'epoch': 1.81}\n",
      "{'loss': 0.0742, 'grad_norm': 4.175767318659525, 'learning_rate': 1.0652557319223988e-06, 'epoch': 1.81}\n",
      "{'loss': 0.0, 'grad_norm': 0.00543892109622625, 'learning_rate': 1.0617283950617285e-06, 'epoch': 1.81}\n",
      "{'loss': 0.0, 'grad_norm': 0.0024397275227007295, 'learning_rate': 1.0582010582010582e-06, 'epoch': 1.81}\n",
      "{'loss': 0.0, 'grad_norm': 9.354979917194169e-09, 'learning_rate': 1.0546737213403881e-06, 'epoch': 1.81}\n",
      "{'loss': 0.0002, 'grad_norm': 0.02888266589632095, 'learning_rate': 1.0511463844797178e-06, 'epoch': 1.81}\n",
      "{'loss': 0.0, 'grad_norm': 0.002458131209002909, 'learning_rate': 1.0476190476190478e-06, 'epoch': 1.81}\n",
      "{'loss': 0.0003, 'grad_norm': 0.03146932607592727, 'learning_rate': 1.0440917107583775e-06, 'epoch': 1.81}\n",
      "{'loss': 0.0001, 'grad_norm': 0.007975430004269563, 'learning_rate': 1.0405643738977072e-06, 'epoch': 1.81}\n",
      "{'loss': 0.0, 'grad_norm': 0.0031402272430444337, 'learning_rate': 1.0370370370370371e-06, 'epoch': 1.82}\n",
      "{'loss': 0.0, 'grad_norm': 0.0005640257748786879, 'learning_rate': 1.0335097001763668e-06, 'epoch': 1.82}\n",
      "{'loss': 0.0, 'grad_norm': 0.005228822080113331, 'learning_rate': 1.0299823633156968e-06, 'epoch': 1.82}\n",
      "{'loss': 0.0, 'grad_norm': 1.1167006155757997e-05, 'learning_rate': 1.0264550264550267e-06, 'epoch': 1.82}\n",
      "{'loss': 0.0, 'grad_norm': 1.6421475992259084e-05, 'learning_rate': 1.0229276895943562e-06, 'epoch': 1.82}\n",
      "{'loss': 0.0, 'grad_norm': 0.00017179282419972096, 'learning_rate': 1.0194003527336861e-06, 'epoch': 1.82}\n",
      "{'loss': 0.0001, 'grad_norm': 0.01047617274611188, 'learning_rate': 1.015873015873016e-06, 'epoch': 1.82}\n",
      "{'loss': 0.0, 'grad_norm': 0.0002617554245571068, 'learning_rate': 1.0123456790123457e-06, 'epoch': 1.82}\n",
      "{'loss': 0.0, 'grad_norm': 0.0008950301415495877, 'learning_rate': 1.0088183421516757e-06, 'epoch': 1.82}\n",
      "{'loss': 0.0211, 'grad_norm': 1.8515516836116028, 'learning_rate': 1.0052910052910054e-06, 'epoch': 1.82}\n",
      "{'loss': 0.0, 'grad_norm': 1.5354085740938843e-05, 'learning_rate': 1.001763668430335e-06, 'epoch': 1.82}\n",
      "{'loss': 0.0, 'grad_norm': 0.0013136252719634905, 'learning_rate': 9.98236331569665e-07, 'epoch': 1.82}\n",
      "{'loss': 0.0001, 'grad_norm': 0.015138884070323854, 'learning_rate': 9.947089947089947e-07, 'epoch': 1.82}\n",
      "{'loss': 0.0, 'grad_norm': 0.001512503037489941, 'learning_rate': 9.911816578483247e-07, 'epoch': 1.82}\n",
      "{'loss': 0.0003, 'grad_norm': 0.03650811251250313, 'learning_rate': 9.876543209876544e-07, 'epoch': 1.82}\n",
      "{'loss': 0.0, 'grad_norm': 0.001724620728698204, 'learning_rate': 9.84126984126984e-07, 'epoch': 1.82}\n",
      "{'loss': 0.0, 'grad_norm': 5.3623202811862104e-05, 'learning_rate': 9.80599647266314e-07, 'epoch': 1.83}\n",
      "{'loss': 0.0, 'grad_norm': 5.938377660144885e-05, 'learning_rate': 9.77072310405644e-07, 'epoch': 1.83}\n",
      "{'loss': 0.0003, 'grad_norm': 0.03539650829994047, 'learning_rate': 9.735449735449736e-07, 'epoch': 1.83}\n",
      "{'loss': 0.0001, 'grad_norm': 0.01362723381038537, 'learning_rate': 9.700176366843034e-07, 'epoch': 1.83}\n",
      "{'loss': 0.0002, 'grad_norm': 0.02170435695583752, 'learning_rate': 9.664902998236333e-07, 'epoch': 1.83}\n",
      "{'loss': 0.0, 'grad_norm': 0.000646161547481818, 'learning_rate': 9.62962962962963e-07, 'epoch': 1.83}\n",
      "{'loss': 0.0, 'grad_norm': 0.004044395940076426, 'learning_rate': 9.59435626102293e-07, 'epoch': 1.83}\n",
      "{'loss': 0.0, 'grad_norm': 0.0015984793505548541, 'learning_rate': 9.559082892416226e-07, 'epoch': 1.83}\n",
      "{'loss': 0.0, 'grad_norm': 0.004539839546766922, 'learning_rate': 9.523809523809525e-07, 'epoch': 1.83}\n",
      "{'loss': 0.0, 'grad_norm': 1.6159201479501806e-05, 'learning_rate': 9.488536155202823e-07, 'epoch': 1.83}\n",
      "{'loss': 0.0, 'grad_norm': 0.0010788009961628672, 'learning_rate': 9.453262786596121e-07, 'epoch': 1.83}\n",
      "{'loss': 0.0, 'grad_norm': 0.0007774529002362729, 'learning_rate': 9.417989417989419e-07, 'epoch': 1.83}\n",
      "{'loss': 0.0001, 'grad_norm': 0.005679728110622766, 'learning_rate': 9.382716049382717e-07, 'epoch': 1.83}\n",
      "{'loss': 0.0, 'grad_norm': 0.0013392319852155708, 'learning_rate': 9.347442680776014e-07, 'epoch': 1.83}\n",
      "{'loss': 0.0, 'grad_norm': 4.983701770885585e-05, 'learning_rate': 9.312169312169313e-07, 'epoch': 1.83}\n",
      "{'loss': 0.0, 'grad_norm': 0.0011072044840184187, 'learning_rate': 9.276895943562611e-07, 'epoch': 1.83}\n",
      "{'loss': 0.0, 'grad_norm': 0.00013566235166995148, 'learning_rate': 9.241622574955909e-07, 'epoch': 1.84}\n",
      "{'loss': 0.0918, 'grad_norm': 7.5205326037808815, 'learning_rate': 9.206349206349208e-07, 'epoch': 1.84}\n",
      "{'loss': 0.0009, 'grad_norm': 0.11087422950857524, 'learning_rate': 9.171075837742504e-07, 'epoch': 1.84}\n",
      "{'loss': 0.0026, 'grad_norm': 0.3988402518914261, 'learning_rate': 9.135802469135802e-07, 'epoch': 1.84}\n",
      "{'loss': 0.0, 'grad_norm': 0.0019477546539639016, 'learning_rate': 9.100529100529102e-07, 'epoch': 1.84}\n",
      "{'loss': 0.0001, 'grad_norm': 0.008947390123494983, 'learning_rate': 9.0652557319224e-07, 'epoch': 1.84}\n",
      "{'loss': 0.0001, 'grad_norm': 0.010916694862046777, 'learning_rate': 9.029982363315698e-07, 'epoch': 1.84}\n",
      "{'loss': 0.0, 'grad_norm': 9.61061593861633e-05, 'learning_rate': 8.994708994708995e-07, 'epoch': 1.84}\n",
      "{'loss': 0.0, 'grad_norm': 7.191758792420599e-06, 'learning_rate': 8.959435626102293e-07, 'epoch': 1.84}\n",
      "{'loss': 0.0045, 'grad_norm': 4.25442790326239, 'learning_rate': 8.924162257495592e-07, 'epoch': 1.84}\n",
      "{'loss': 0.0, 'grad_norm': 0.000996616411078241, 'learning_rate': 8.88888888888889e-07, 'epoch': 1.84}\n",
      "{'loss': 0.0, 'grad_norm': 0.00017841616735292574, 'learning_rate': 8.853615520282188e-07, 'epoch': 1.84}\n",
      "{'loss': 0.0, 'grad_norm': 3.089755254834021e-08, 'learning_rate': 8.818342151675485e-07, 'epoch': 1.84}\n",
      "{'loss': 0.0, 'grad_norm': 0.00017594547073467007, 'learning_rate': 8.783068783068783e-07, 'epoch': 1.84}\n",
      "{'loss': 0.0, 'grad_norm': 0.0006721895256610928, 'learning_rate': 8.747795414462081e-07, 'epoch': 1.84}\n",
      "{'loss': 0.0, 'grad_norm': 0.0015120291249728966, 'learning_rate': 8.712522045855381e-07, 'epoch': 1.85}\n",
      "{'loss': 0.0001, 'grad_norm': 0.008474597888645548, 'learning_rate': 8.677248677248679e-07, 'epoch': 1.85}\n",
      "{'loss': 0.0, 'grad_norm': 0.0010425233983906508, 'learning_rate': 8.641975308641976e-07, 'epoch': 1.85}\n",
      "{'loss': 0.0, 'grad_norm': 1.3542894650866333e-05, 'learning_rate': 8.606701940035274e-07, 'epoch': 1.85}\n",
      "{'loss': 0.0001, 'grad_norm': 0.009197157974584253, 'learning_rate': 8.571428571428572e-07, 'epoch': 1.85}\n",
      "{'loss': 0.0, 'grad_norm': 0.00253423908357895, 'learning_rate': 8.536155202821871e-07, 'epoch': 1.85}\n",
      "{'loss': 0.0, 'grad_norm': 1.0008493482325089e-05, 'learning_rate': 8.500881834215169e-07, 'epoch': 1.85}\n",
      "{'loss': 0.0, 'grad_norm': 0.0003718397057159748, 'learning_rate': 8.465608465608466e-07, 'epoch': 1.85}\n",
      "{'loss': 0.0, 'grad_norm': 0.001465245250226325, 'learning_rate': 8.430335097001764e-07, 'epoch': 1.85}\n",
      "{'loss': 0.0002, 'grad_norm': 0.015658149721050514, 'learning_rate': 8.395061728395062e-07, 'epoch': 1.85}\n",
      "{'loss': 0.0001, 'grad_norm': 0.012402893766650006, 'learning_rate': 8.35978835978836e-07, 'epoch': 1.85}\n",
      "{'loss': 0.0, 'grad_norm': 0.0033642191289496503, 'learning_rate': 8.324514991181659e-07, 'epoch': 1.85}\n",
      "{'loss': 0.0339, 'grad_norm': 3.6744046202554355, 'learning_rate': 8.289241622574956e-07, 'epoch': 1.85}\n",
      "{'loss': 0.0038, 'grad_norm': 0.5441794311093471, 'learning_rate': 8.253968253968254e-07, 'epoch': 1.85}\n",
      "{'loss': 0.0, 'grad_norm': 0.004610843024047733, 'learning_rate': 8.218694885361553e-07, 'epoch': 1.85}\n",
      "{'loss': 0.0, 'grad_norm': 0.00013129242569294306, 'learning_rate': 8.183421516754851e-07, 'epoch': 1.85}\n",
      "{'loss': 0.0, 'grad_norm': 4.319876699599572e-05, 'learning_rate': 8.14814814814815e-07, 'epoch': 1.86}\n",
      "{'loss': 0.0002, 'grad_norm': 0.018323974454357655, 'learning_rate': 8.112874779541447e-07, 'epoch': 1.86}\n",
      "{'loss': 0.0, 'grad_norm': 6.6328584536584246e-06, 'learning_rate': 8.077601410934745e-07, 'epoch': 1.86}\n",
      "{'loss': 0.0, 'grad_norm': 0.0018823512069560027, 'learning_rate': 8.042328042328043e-07, 'epoch': 1.86}\n",
      "{'loss': 0.0, 'grad_norm': 0.005532251146599437, 'learning_rate': 8.007054673721341e-07, 'epoch': 1.86}\n",
      "{'loss': 0.0, 'grad_norm': 5.703274363170379e-06, 'learning_rate': 7.971781305114639e-07, 'epoch': 1.86}\n",
      "{'loss': 0.0013, 'grad_norm': 0.18239507580426817, 'learning_rate': 7.936507936507937e-07, 'epoch': 1.86}\n",
      "{'loss': 0.0, 'grad_norm': 7.93484035227323e-06, 'learning_rate': 7.901234567901235e-07, 'epoch': 1.86}\n",
      "{'loss': 0.0, 'grad_norm': 0.0010439400816008328, 'learning_rate': 7.865961199294533e-07, 'epoch': 1.86}\n",
      "{'loss': 0.0005, 'grad_norm': 0.039516967282475945, 'learning_rate': 7.830687830687832e-07, 'epoch': 1.86}\n",
      "{'loss': 0.0003, 'grad_norm': 0.03233101563000936, 'learning_rate': 7.79541446208113e-07, 'epoch': 1.86}\n",
      "{'loss': 0.0, 'grad_norm': 0.0001329464599384787, 'learning_rate': 7.760141093474428e-07, 'epoch': 1.86}\n",
      "{'loss': 0.0, 'grad_norm': 3.063616999944368e-06, 'learning_rate': 7.724867724867726e-07, 'epoch': 1.86}\n",
      "{'loss': 0.0, 'grad_norm': 0.0014340698172889662, 'learning_rate': 7.689594356261024e-07, 'epoch': 1.86}\n",
      "{'loss': 0.0, 'grad_norm': 0.0007866328995341595, 'learning_rate': 7.654320987654322e-07, 'epoch': 1.86}\n",
      "{'loss': 0.0048, 'grad_norm': 0.46627622162461146, 'learning_rate': 7.61904761904762e-07, 'epoch': 1.86}\n",
      "{'loss': 0.0, 'grad_norm': 3.2824854888544735e-06, 'learning_rate': 7.583774250440917e-07, 'epoch': 1.87}\n",
      "{'loss': 0.0, 'grad_norm': 0.0006863793256192384, 'learning_rate': 7.548500881834216e-07, 'epoch': 1.87}\n",
      "{'loss': 0.0, 'grad_norm': 0.0008268453601300114, 'learning_rate': 7.513227513227514e-07, 'epoch': 1.87}\n",
      "{'loss': 0.0, 'grad_norm': 0.0024589801069971854, 'learning_rate': 7.477954144620812e-07, 'epoch': 1.87}\n",
      "{'loss': 0.0, 'grad_norm': 0.00026882140172834, 'learning_rate': 7.442680776014109e-07, 'epoch': 1.87}\n",
      "{'loss': 0.0, 'grad_norm': 0.0004419868284388884, 'learning_rate': 7.407407407407407e-07, 'epoch': 1.87}\n",
      "{'loss': 0.0, 'grad_norm': 0.001414840132355681, 'learning_rate': 7.372134038800705e-07, 'epoch': 1.87}\n",
      "{'loss': 0.0, 'grad_norm': 0.0002180039454875338, 'learning_rate': 7.336860670194005e-07, 'epoch': 1.87}\n",
      "{'loss': 0.0, 'grad_norm': 8.521518493531527e-05, 'learning_rate': 7.301587301587303e-07, 'epoch': 1.87}\n",
      "{'loss': 0.0, 'grad_norm': 2.1050243910698632e-05, 'learning_rate': 7.2663139329806e-07, 'epoch': 1.87}\n",
      "{'loss': 0.0, 'grad_norm': 0.00018412234633779777, 'learning_rate': 7.231040564373898e-07, 'epoch': 1.87}\n",
      "{'loss': 0.0, 'grad_norm': 0.0006558196111335734, 'learning_rate': 7.195767195767196e-07, 'epoch': 1.87}\n",
      "{'loss': 0.0001, 'grad_norm': 0.009911552907446183, 'learning_rate': 7.160493827160495e-07, 'epoch': 1.87}\n",
      "{'loss': 0.0, 'grad_norm': 0.0024849672082663666, 'learning_rate': 7.125220458553793e-07, 'epoch': 1.87}\n",
      "{'loss': 0.0, 'grad_norm': 9.092756448184095e-05, 'learning_rate': 7.08994708994709e-07, 'epoch': 1.87}\n",
      "{'loss': 0.0, 'grad_norm': 0.004436141325712003, 'learning_rate': 7.054673721340388e-07, 'epoch': 1.87}\n",
      "{'loss': 0.0, 'grad_norm': 0.00016417010654742173, 'learning_rate': 7.019400352733686e-07, 'epoch': 1.88}\n",
      "{'loss': 0.0002, 'grad_norm': 0.020027401150667118, 'learning_rate': 6.984126984126984e-07, 'epoch': 1.88}\n",
      "{'loss': 0.0003, 'grad_norm': 0.03262111239032489, 'learning_rate': 6.948853615520284e-07, 'epoch': 1.88}\n",
      "{'loss': 0.0, 'grad_norm': 0.00138312205109877, 'learning_rate': 6.91358024691358e-07, 'epoch': 1.88}\n",
      "{'loss': 0.0, 'grad_norm': 4.143620675154094e-07, 'learning_rate': 6.878306878306879e-07, 'epoch': 1.88}\n",
      "{'loss': 0.0, 'grad_norm': 0.00048091446104581484, 'learning_rate': 6.843033509700177e-07, 'epoch': 1.88}\n",
      "{'loss': 0.0, 'grad_norm': 0.0014008486455082307, 'learning_rate': 6.807760141093475e-07, 'epoch': 1.88}\n",
      "{'loss': 0.0, 'grad_norm': 3.5140223736778088e-06, 'learning_rate': 6.772486772486774e-07, 'epoch': 1.88}\n",
      "{'loss': 0.0003, 'grad_norm': 0.023704009454032976, 'learning_rate': 6.737213403880071e-07, 'epoch': 1.88}\n",
      "{'loss': 0.0, 'grad_norm': 0.0006480994118300781, 'learning_rate': 6.701940035273369e-07, 'epoch': 1.88}\n",
      "{'loss': 0.5947, 'grad_norm': 22.19695482418246, 'learning_rate': 6.666666666666667e-07, 'epoch': 1.88}\n",
      "{'loss': 0.0, 'grad_norm': 0.00028367308902580037, 'learning_rate': 6.631393298059965e-07, 'epoch': 1.88}\n",
      "{'loss': 0.0493, 'grad_norm': 3.0678233377157014, 'learning_rate': 6.596119929453263e-07, 'epoch': 1.88}\n",
      "{'loss': 0.0, 'grad_norm': 3.937198531954759e-05, 'learning_rate': 6.560846560846561e-07, 'epoch': 1.88}\n",
      "{'loss': 0.001, 'grad_norm': 0.12981481851813162, 'learning_rate': 6.525573192239859e-07, 'epoch': 1.88}\n",
      "{'loss': 0.0, 'grad_norm': 0.000931824726151179, 'learning_rate': 6.490299823633157e-07, 'epoch': 1.89}\n",
      "{'loss': 0.0, 'grad_norm': 4.1076087469488374e-06, 'learning_rate': 6.455026455026456e-07, 'epoch': 1.89}\n",
      "{'loss': 0.0762, 'grad_norm': 6.15755607576349, 'learning_rate': 6.419753086419754e-07, 'epoch': 1.89}\n",
      "{'loss': 0.0007, 'grad_norm': 0.08136850527219583, 'learning_rate': 6.384479717813052e-07, 'epoch': 1.89}\n",
      "{'loss': 0.144, 'grad_norm': 9.185813697153268, 'learning_rate': 6.34920634920635e-07, 'epoch': 1.89}\n",
      "{'loss': 0.0, 'grad_norm': 0.0010369735372036655, 'learning_rate': 6.313932980599648e-07, 'epoch': 1.89}\n",
      "{'loss': 0.0, 'grad_norm': 7.170906657584714e-05, 'learning_rate': 6.278659611992946e-07, 'epoch': 1.89}\n",
      "{'loss': 0.0015, 'grad_norm': 0.21180759339490057, 'learning_rate': 6.243386243386243e-07, 'epoch': 1.89}\n",
      "{'loss': 0.0154, 'grad_norm': 1.8968492936681618, 'learning_rate': 6.208112874779542e-07, 'epoch': 1.89}\n",
      "{'loss': 0.0, 'grad_norm': 0.0010693154871617577, 'learning_rate': 6.17283950617284e-07, 'epoch': 1.89}\n",
      "{'loss': 0.0, 'grad_norm': 0.0009593358373889379, 'learning_rate': 6.137566137566138e-07, 'epoch': 1.89}\n",
      "{'loss': 0.0, 'grad_norm': 0.005423520264046245, 'learning_rate': 6.102292768959436e-07, 'epoch': 1.89}\n",
      "{'loss': 0.0, 'grad_norm': 0.00011931902707200133, 'learning_rate': 6.067019400352734e-07, 'epoch': 1.89}\n",
      "{'loss': 0.0004, 'grad_norm': 0.03432353947016045, 'learning_rate': 6.031746031746032e-07, 'epoch': 1.89}\n",
      "{'loss': 0.0005, 'grad_norm': 0.060848628170751165, 'learning_rate': 5.996472663139329e-07, 'epoch': 1.89}\n",
      "{'loss': 0.0434, 'grad_norm': 3.995573097535144, 'learning_rate': 5.961199294532629e-07, 'epoch': 1.89}\n",
      "{'loss': 0.0195, 'grad_norm': 3.337184128118834, 'learning_rate': 5.925925925925927e-07, 'epoch': 1.9}\n",
      "{'loss': 0.0, 'grad_norm': 1.3346930133281044e-05, 'learning_rate': 5.890652557319224e-07, 'epoch': 1.9}\n",
      "{'loss': 0.0, 'grad_norm': 0.0001473744338507368, 'learning_rate': 5.855379188712522e-07, 'epoch': 1.9}\n",
      "{'loss': 0.0, 'grad_norm': 3.873381448388782e-06, 'learning_rate': 5.82010582010582e-07, 'epoch': 1.9}\n",
      "{'loss': 0.0001, 'grad_norm': 0.017951695399234854, 'learning_rate': 5.784832451499119e-07, 'epoch': 1.9}\n",
      "{'loss': 0.0002, 'grad_norm': 0.019404340236655687, 'learning_rate': 5.749559082892417e-07, 'epoch': 1.9}\n",
      "{'loss': 0.0, 'grad_norm': 5.123999525867746e-06, 'learning_rate': 5.714285714285715e-07, 'epoch': 1.9}\n",
      "{'loss': 0.0, 'grad_norm': 3.511301099363453e-05, 'learning_rate': 5.679012345679013e-07, 'epoch': 1.9}\n",
      "{'loss': 0.0, 'grad_norm': 2.046743990479607e-05, 'learning_rate': 5.64373897707231e-07, 'epoch': 1.9}\n",
      "{'loss': 0.0001, 'grad_norm': 0.013190673763733582, 'learning_rate': 5.608465608465608e-07, 'epoch': 1.9}\n",
      "{'loss': 0.0, 'grad_norm': 2.7010975135940436e-07, 'learning_rate': 5.573192239858908e-07, 'epoch': 1.9}\n",
      "{'loss': 0.0, 'grad_norm': 0.002546070566202859, 'learning_rate': 5.537918871252205e-07, 'epoch': 1.9}\n",
      "{'loss': 0.0002, 'grad_norm': 0.018173035205145027, 'learning_rate': 5.502645502645503e-07, 'epoch': 1.9}\n",
      "{'loss': 0.0, 'grad_norm': 0.0004747228238415261, 'learning_rate': 5.467372134038801e-07, 'epoch': 1.9}\n",
      "{'loss': 0.0065, 'grad_norm': 0.7283245173501952, 'learning_rate': 5.432098765432099e-07, 'epoch': 1.9}\n",
      "{'loss': 0.0, 'grad_norm': 3.2371688097277066e-05, 'learning_rate': 5.396825396825398e-07, 'epoch': 1.9}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 95%|█████████▌| 3000/3150 [12:29<00:34,  4.40it/s]12/23/2024 06:47:36 - INFO - FlagEmbedding.finetune.embedder.encoder_only.base.trainer -   Saving model checkpoint to ./test_encoder_only_base_bge-large-en-v1.5/checkpoint-3000\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 0.0684, 'grad_norm': 7.017930091803245, 'learning_rate': 5.361552028218695e-07, 'epoch': 1.91}\n",
      "{'loss': 0.0, 'grad_norm': 0.000511339045166218, 'learning_rate': 5.326278659611994e-07, 'epoch': 1.91}\n",
      "{'loss': 0.0, 'grad_norm': 0.0005950045300509824, 'learning_rate': 5.291005291005291e-07, 'epoch': 1.91}\n",
      "{'loss': 0.0, 'grad_norm': 0.003924217893271839, 'learning_rate': 5.255731922398589e-07, 'epoch': 1.91}\n",
      "{'loss': 0.0, 'grad_norm': 0.004307797663678439, 'learning_rate': 5.220458553791887e-07, 'epoch': 1.91}\n",
      "{'loss': 0.0, 'grad_norm': 2.603897655768594e-05, 'learning_rate': 5.185185185185186e-07, 'epoch': 1.91}\n",
      "{'loss': 0.0, 'grad_norm': 5.483371389463175e-06, 'learning_rate': 5.149911816578484e-07, 'epoch': 1.91}\n",
      "{'loss': 0.4844, 'grad_norm': 20.3684123804579, 'learning_rate': 5.114638447971781e-07, 'epoch': 1.91}\n",
      "{'loss': 0.0001, 'grad_norm': 0.01247325742979955, 'learning_rate': 5.07936507936508e-07, 'epoch': 1.91}\n",
      "{'loss': 0.0004, 'grad_norm': 0.05214513400326202, 'learning_rate': 5.044091710758378e-07, 'epoch': 1.91}\n",
      "{'loss': 0.0043, 'grad_norm': 0.39754653871157175, 'learning_rate': 5.008818342151675e-07, 'epoch': 1.91}\n",
      "{'loss': 0.0, 'grad_norm': 0.0011053969847313325, 'learning_rate': 4.973544973544974e-07, 'epoch': 1.91}\n",
      "{'loss': 0.0, 'grad_norm': 4.441425509402899e-06, 'learning_rate': 4.938271604938272e-07, 'epoch': 1.91}\n",
      "{'loss': 0.0, 'grad_norm': 4.610890750135553e-05, 'learning_rate': 4.90299823633157e-07, 'epoch': 1.91}\n",
      "{'loss': 0.0, 'grad_norm': 3.9493094037863396e-05, 'learning_rate': 4.867724867724868e-07, 'epoch': 1.91}\n",
      "{'loss': 0.0, 'grad_norm': 0.00039020145079620436, 'learning_rate': 4.832451499118166e-07, 'epoch': 1.91}\n",
      "{'loss': 0.0144, 'grad_norm': 1.88804343023699, 'learning_rate': 4.797178130511465e-07, 'epoch': 1.92}\n",
      "{'loss': 0.0, 'grad_norm': 0.003097977659349139, 'learning_rate': 4.7619047619047623e-07, 'epoch': 1.92}\n",
      "{'loss': 0.0, 'grad_norm': 0.0003778866036213471, 'learning_rate': 4.7266313932980605e-07, 'epoch': 1.92}\n",
      "{'loss': 0.0, 'grad_norm': 5.828519628154781e-05, 'learning_rate': 4.6913580246913586e-07, 'epoch': 1.92}\n",
      "{'loss': 0.0003, 'grad_norm': 0.044727839322584735, 'learning_rate': 4.6560846560846563e-07, 'epoch': 1.92}\n",
      "{'loss': 0.0, 'grad_norm': 0.0005428033173977267, 'learning_rate': 4.6208112874779545e-07, 'epoch': 1.92}\n",
      "{'loss': 0.0, 'grad_norm': 4.7212207596064364e-05, 'learning_rate': 4.585537918871252e-07, 'epoch': 1.92}\n",
      "{'loss': 0.0005, 'grad_norm': 0.05847935437065705, 'learning_rate': 4.550264550264551e-07, 'epoch': 1.92}\n",
      "{'loss': 0.0001, 'grad_norm': 0.011699597532878268, 'learning_rate': 4.514991181657849e-07, 'epoch': 1.92}\n",
      "{'loss': 0.0002, 'grad_norm': 0.01870368982276248, 'learning_rate': 4.4797178130511467e-07, 'epoch': 1.92}\n",
      "{'loss': 0.0002, 'grad_norm': 0.015383468039619994, 'learning_rate': 4.444444444444445e-07, 'epoch': 1.92}\n",
      "{'loss': 0.0, 'grad_norm': 0.00011209545661644016, 'learning_rate': 4.4091710758377425e-07, 'epoch': 1.92}\n",
      "{'loss': 0.0, 'grad_norm': 0.003868200838857486, 'learning_rate': 4.3738977072310407e-07, 'epoch': 1.92}\n",
      "{'loss': 0.0, 'grad_norm': 2.60775052300705e-06, 'learning_rate': 4.3386243386243395e-07, 'epoch': 1.92}\n",
      "{'loss': 0.0, 'grad_norm': 7.593150131535777e-05, 'learning_rate': 4.303350970017637e-07, 'epoch': 1.92}\n",
      "{'loss': 0.068, 'grad_norm': 5.66971969638243, 'learning_rate': 4.2680776014109353e-07, 'epoch': 1.93}\n",
      "{'loss': 0.0, 'grad_norm': 1.1517619156772149e-06, 'learning_rate': 4.232804232804233e-07, 'epoch': 1.93}\n",
      "{'loss': 0.0, 'grad_norm': 0.00016261318716400273, 'learning_rate': 4.197530864197531e-07, 'epoch': 1.93}\n",
      "{'loss': 0.0, 'grad_norm': 9.934423908518212e-05, 'learning_rate': 4.1622574955908293e-07, 'epoch': 1.93}\n",
      "{'loss': 0.0, 'grad_norm': 4.630650865015419e-05, 'learning_rate': 4.126984126984127e-07, 'epoch': 1.93}\n",
      "{'loss': 0.0, 'grad_norm': 0.0002713659199822411, 'learning_rate': 4.0917107583774257e-07, 'epoch': 1.93}\n",
      "{'loss': 0.0, 'grad_norm': 9.59855588838112e-05, 'learning_rate': 4.0564373897707234e-07, 'epoch': 1.93}\n",
      "{'loss': 0.0, 'grad_norm': 0.00025703915471810755, 'learning_rate': 4.0211640211640215e-07, 'epoch': 1.93}\n",
      "{'loss': 0.0001, 'grad_norm': 0.008073813296251653, 'learning_rate': 3.9858906525573197e-07, 'epoch': 1.93}\n",
      "{'loss': 0.0, 'grad_norm': 0.0003134547688797712, 'learning_rate': 3.9506172839506174e-07, 'epoch': 1.93}\n",
      "{'loss': 0.0, 'grad_norm': 0.003652804569021769, 'learning_rate': 3.915343915343916e-07, 'epoch': 1.93}\n",
      "{'loss': 0.0, 'grad_norm': 0.0004813861229114684, 'learning_rate': 3.880070546737214e-07, 'epoch': 1.93}\n",
      "{'loss': 0.0, 'grad_norm': 0.006043893355093694, 'learning_rate': 3.844797178130512e-07, 'epoch': 1.93}\n",
      "{'loss': 0.0001, 'grad_norm': 0.0161769819762257, 'learning_rate': 3.80952380952381e-07, 'epoch': 1.93}\n",
      "{'loss': 0.0006, 'grad_norm': 0.06428789504253596, 'learning_rate': 3.774250440917108e-07, 'epoch': 1.93}\n",
      "{'loss': 0.0, 'grad_norm': 0.0016096208795644844, 'learning_rate': 3.738977072310406e-07, 'epoch': 1.93}\n",
      "{'loss': 0.0001, 'grad_norm': 0.006103003897672634, 'learning_rate': 3.7037037037037036e-07, 'epoch': 1.94}\n",
      "{'loss': 0.0005, 'grad_norm': 0.07220813036353753, 'learning_rate': 3.6684303350970024e-07, 'epoch': 1.94}\n",
      "{'loss': 0.0, 'grad_norm': 7.71015772480006e-05, 'learning_rate': 3.6331569664903e-07, 'epoch': 1.94}\n",
      "{'loss': 0.0, 'grad_norm': 2.5767988910071672e-05, 'learning_rate': 3.597883597883598e-07, 'epoch': 1.94}\n",
      "{'loss': 0.0, 'grad_norm': 0.0021044919371881357, 'learning_rate': 3.5626102292768964e-07, 'epoch': 1.94}\n",
      "{'loss': 0.0001, 'grad_norm': 0.0067619915429907, 'learning_rate': 3.527336860670194e-07, 'epoch': 1.94}\n",
      "{'loss': 0.0, 'grad_norm': 0.0006362331901673785, 'learning_rate': 3.492063492063492e-07, 'epoch': 1.94}\n",
      "{'loss': 0.0, 'grad_norm': 0.002150033841525042, 'learning_rate': 3.45679012345679e-07, 'epoch': 1.94}\n",
      "{'loss': 0.0, 'grad_norm': 2.047805190423887e-05, 'learning_rate': 3.4215167548500886e-07, 'epoch': 1.94}\n",
      "{'loss': 0.0, 'grad_norm': 4.025536354708105e-05, 'learning_rate': 3.386243386243387e-07, 'epoch': 1.94}\n",
      "{'loss': 0.0, 'grad_norm': 1.4238546860667142e-05, 'learning_rate': 3.3509700176366844e-07, 'epoch': 1.94}\n",
      "{'loss': 0.0, 'grad_norm': 1.937328256576442e-06, 'learning_rate': 3.3156966490299826e-07, 'epoch': 1.94}\n",
      "{'loss': 0.0, 'grad_norm': 3.171681619058705e-05, 'learning_rate': 3.2804232804232803e-07, 'epoch': 1.94}\n",
      "{'loss': 0.0001, 'grad_norm': 0.023320799104447645, 'learning_rate': 3.2451499118165785e-07, 'epoch': 1.94}\n",
      "{'loss': 0.0, 'grad_norm': 7.934331823314003e-06, 'learning_rate': 3.209876543209877e-07, 'epoch': 1.94}\n",
      "{'loss': 0.0001, 'grad_norm': 0.00799464991734567, 'learning_rate': 3.174603174603175e-07, 'epoch': 1.94}\n",
      "{'loss': 0.0, 'grad_norm': 0.0012578571984434596, 'learning_rate': 3.139329805996473e-07, 'epoch': 1.95}\n",
      "{'loss': 0.0, 'grad_norm': 7.72411087198512e-05, 'learning_rate': 3.104056437389771e-07, 'epoch': 1.95}\n",
      "{'loss': 0.0002, 'grad_norm': 0.022714740994025026, 'learning_rate': 3.068783068783069e-07, 'epoch': 1.95}\n",
      "{'loss': 0.0, 'grad_norm': 0.0015504349750055965, 'learning_rate': 3.033509700176367e-07, 'epoch': 1.95}\n",
      "{'loss': 0.0, 'grad_norm': 0.0018021214917382257, 'learning_rate': 2.9982363315696647e-07, 'epoch': 1.95}\n",
      "{'loss': 0.0, 'grad_norm': 0.004113905169376137, 'learning_rate': 2.9629629629629634e-07, 'epoch': 1.95}\n",
      "{'loss': 0.0015, 'grad_norm': 0.2016581149609865, 'learning_rate': 2.927689594356261e-07, 'epoch': 1.95}\n",
      "{'loss': 0.0, 'grad_norm': 0.00018449593141463475, 'learning_rate': 2.8924162257495593e-07, 'epoch': 1.95}\n",
      "{'loss': 0.0001, 'grad_norm': 0.008245391589446278, 'learning_rate': 2.8571428571428575e-07, 'epoch': 1.95}\n",
      "{'loss': 0.0323, 'grad_norm': 3.780273689776535, 'learning_rate': 2.821869488536155e-07, 'epoch': 1.95}\n",
      "{'loss': 0.0001, 'grad_norm': 0.00852297706812535, 'learning_rate': 2.786596119929454e-07, 'epoch': 1.95}\n",
      "{'loss': 0.0, 'grad_norm': 0.0018313366702628945, 'learning_rate': 2.7513227513227515e-07, 'epoch': 1.95}\n",
      "{'loss': 0.0, 'grad_norm': 0.00011691011588432173, 'learning_rate': 2.7160493827160497e-07, 'epoch': 1.95}\n",
      "{'loss': 0.0, 'grad_norm': 0.0016125895986981674, 'learning_rate': 2.6807760141093473e-07, 'epoch': 1.95}\n",
      "{'loss': 0.0, 'grad_norm': 8.771859359620397e-05, 'learning_rate': 2.6455026455026455e-07, 'epoch': 1.95}\n",
      "{'loss': 0.0, 'grad_norm': 4.0505952256235e-06, 'learning_rate': 2.6102292768959437e-07, 'epoch': 1.95}\n",
      "{'loss': 0.0, 'grad_norm': 2.8934357866479634e-06, 'learning_rate': 2.574955908289242e-07, 'epoch': 1.96}\n",
      "{'loss': 0.0, 'grad_norm': 3.0393639722872672e-05, 'learning_rate': 2.53968253968254e-07, 'epoch': 1.96}\n",
      "{'loss': 0.0, 'grad_norm': 0.0001197429164920675, 'learning_rate': 2.504409171075838e-07, 'epoch': 1.96}\n",
      "{'loss': 0.0003, 'grad_norm': 0.033777332957387246, 'learning_rate': 2.469135802469136e-07, 'epoch': 1.96}\n",
      "{'loss': 0.131, 'grad_norm': 6.288003052747146, 'learning_rate': 2.433862433862434e-07, 'epoch': 1.96}\n",
      "{'loss': 0.0001, 'grad_norm': 0.012765237913542997, 'learning_rate': 2.3985890652557323e-07, 'epoch': 1.96}\n",
      "{'loss': 0.0, 'grad_norm': 0.00025312920024049074, 'learning_rate': 2.3633156966490302e-07, 'epoch': 1.96}\n",
      "{'loss': 0.0157, 'grad_norm': 1.3197526817113876, 'learning_rate': 2.3280423280423281e-07, 'epoch': 1.96}\n",
      "{'loss': 0.0006, 'grad_norm': 0.08167493443773247, 'learning_rate': 2.292768959435626e-07, 'epoch': 1.96}\n",
      "{'loss': 0.8901, 'grad_norm': 35.861065115399704, 'learning_rate': 2.2574955908289245e-07, 'epoch': 1.96}\n",
      "{'loss': 0.0, 'grad_norm': 0.001171426253310791, 'learning_rate': 2.2222222222222224e-07, 'epoch': 1.96}\n",
      "{'loss': 0.0, 'grad_norm': 0.000942317606516971, 'learning_rate': 2.1869488536155204e-07, 'epoch': 1.96}\n",
      "{'loss': 0.0009, 'grad_norm': 0.12471578368657384, 'learning_rate': 2.1516754850088186e-07, 'epoch': 1.96}\n",
      "{'loss': 0.0, 'grad_norm': 2.4370120235734013e-05, 'learning_rate': 2.1164021164021165e-07, 'epoch': 1.96}\n",
      "{'loss': 0.0, 'grad_norm': 0.003842534065516292, 'learning_rate': 2.0811287477954147e-07, 'epoch': 1.96}\n",
      "{'loss': 0.0, 'grad_norm': 0.0002117079538596778, 'learning_rate': 2.0458553791887128e-07, 'epoch': 1.97}\n",
      "{'loss': 0.0, 'grad_norm': 0.004507337231824404, 'learning_rate': 2.0105820105820108e-07, 'epoch': 1.97}\n",
      "{'loss': 0.0123, 'grad_norm': 2.0972165689399205, 'learning_rate': 1.9753086419753087e-07, 'epoch': 1.97}\n",
      "{'loss': 0.0, 'grad_norm': 0.0009499857021245386, 'learning_rate': 1.940035273368607e-07, 'epoch': 1.97}\n",
      "{'loss': 0.0001, 'grad_norm': 0.011007416869556516, 'learning_rate': 1.904761904761905e-07, 'epoch': 1.97}\n",
      "{'loss': 0.0, 'grad_norm': 0.00031314452693028853, 'learning_rate': 1.869488536155203e-07, 'epoch': 1.97}\n",
      "{'loss': 0.0406, 'grad_norm': 3.133294808035934, 'learning_rate': 1.8342151675485012e-07, 'epoch': 1.97}\n",
      "{'loss': 0.0, 'grad_norm': 0.00012721737014292995, 'learning_rate': 1.798941798941799e-07, 'epoch': 1.97}\n",
      "{'loss': 0.0, 'grad_norm': 0.0012331950656217505, 'learning_rate': 1.763668430335097e-07, 'epoch': 1.97}\n",
      "{'loss': 0.0, 'grad_norm': 0.00019371085895894088, 'learning_rate': 1.728395061728395e-07, 'epoch': 1.97}\n",
      "{'loss': 0.0, 'grad_norm': 0.0006876610573984653, 'learning_rate': 1.6931216931216934e-07, 'epoch': 1.97}\n",
      "{'loss': 0.0, 'grad_norm': 0.003043942876051606, 'learning_rate': 1.6578483245149913e-07, 'epoch': 1.97}\n",
      "{'loss': 0.0, 'grad_norm': 5.570820710691136e-06, 'learning_rate': 1.6225749559082892e-07, 'epoch': 1.97}\n",
      "{'loss': 0.0084, 'grad_norm': 0.9241918771146999, 'learning_rate': 1.5873015873015874e-07, 'epoch': 1.97}\n",
      "{'loss': 0.0001, 'grad_norm': 0.010668706113787582, 'learning_rate': 1.5520282186948856e-07, 'epoch': 1.97}\n",
      "{'loss': 0.0, 'grad_norm': 9.937013161407782e-05, 'learning_rate': 1.5167548500881835e-07, 'epoch': 1.97}\n",
      "{'loss': 0.0, 'grad_norm': 1.0460151476522579e-05, 'learning_rate': 1.4814814814814817e-07, 'epoch': 1.98}\n",
      "{'loss': 0.0, 'grad_norm': 3.4911135821302845e-05, 'learning_rate': 1.4462081128747796e-07, 'epoch': 1.98}\n",
      "{'loss': 0.0, 'grad_norm': 6.265151004740304e-05, 'learning_rate': 1.4109347442680776e-07, 'epoch': 1.98}\n",
      "{'loss': 0.0, 'grad_norm': 0.00022453282503102275, 'learning_rate': 1.3756613756613757e-07, 'epoch': 1.98}\n",
      "{'loss': 0.0, 'grad_norm': 7.574338067568745e-05, 'learning_rate': 1.3403880070546737e-07, 'epoch': 1.98}\n",
      "{'loss': 0.0, 'grad_norm': 6.268764998486763e-07, 'learning_rate': 1.3051146384479719e-07, 'epoch': 1.98}\n",
      "{'loss': 0.0, 'grad_norm': 5.10557255367733e-07, 'learning_rate': 1.26984126984127e-07, 'epoch': 1.98}\n",
      "{'loss': 0.0, 'grad_norm': 2.8119808849843654e-06, 'learning_rate': 1.234567901234568e-07, 'epoch': 1.98}\n",
      "{'loss': 0.0003, 'grad_norm': 0.02859943745838939, 'learning_rate': 1.1992945326278662e-07, 'epoch': 1.98}\n",
      "{'loss': 0.0, 'grad_norm': 4.0597290030787186e-05, 'learning_rate': 1.1640211640211641e-07, 'epoch': 1.98}\n",
      "{'loss': 0.0, 'grad_norm': 0.0010799397916091836, 'learning_rate': 1.1287477954144623e-07, 'epoch': 1.98}\n",
      "{'loss': 0.0015, 'grad_norm': 0.2170854829735469, 'learning_rate': 1.0934744268077602e-07, 'epoch': 1.98}\n",
      "{'loss': 0.0, 'grad_norm': 0.0024801494725132617, 'learning_rate': 1.0582010582010582e-07, 'epoch': 1.98}\n",
      "{'loss': 0.0, 'grad_norm': 3.4255744029830784e-05, 'learning_rate': 1.0229276895943564e-07, 'epoch': 1.98}\n",
      "{'loss': 0.0, 'grad_norm': 0.00039512727798635556, 'learning_rate': 9.876543209876543e-08, 'epoch': 1.98}\n",
      "{'loss': 0.0, 'grad_norm': 0.0006474007929400392, 'learning_rate': 9.523809523809525e-08, 'epoch': 1.98}\n",
      "{'loss': 0.4368, 'grad_norm': 20.33885384750277, 'learning_rate': 9.171075837742506e-08, 'epoch': 1.99}\n",
      "{'loss': 0.0006, 'grad_norm': 0.07245427394056331, 'learning_rate': 8.818342151675485e-08, 'epoch': 1.99}\n",
      "{'loss': 0.0, 'grad_norm': 9.34553454968883e-05, 'learning_rate': 8.465608465608467e-08, 'epoch': 1.99}\n",
      "{'loss': 0.0, 'grad_norm': 0.0011323300301301217, 'learning_rate': 8.112874779541446e-08, 'epoch': 1.99}\n",
      "{'loss': 0.0016, 'grad_norm': 0.19478936708252317, 'learning_rate': 7.760141093474428e-08, 'epoch': 1.99}\n",
      "{'loss': 0.0049, 'grad_norm': 0.7588467177158765, 'learning_rate': 7.407407407407409e-08, 'epoch': 1.99}\n",
      "{'loss': 0.0, 'grad_norm': 0.0009042426237441818, 'learning_rate': 7.054673721340388e-08, 'epoch': 1.99}\n",
      "{'loss': 0.0, 'grad_norm': 5.4026680880987734e-05, 'learning_rate': 6.701940035273368e-08, 'epoch': 1.99}\n",
      "{'loss': 0.0, 'grad_norm': 4.146444168986008e-05, 'learning_rate': 6.34920634920635e-08, 'epoch': 1.99}\n",
      "{'loss': 0.0, 'grad_norm': 5.293171053988699e-07, 'learning_rate': 5.996472663139331e-08, 'epoch': 1.99}\n",
      "{'loss': 0.0262, 'grad_norm': 3.837932370956164, 'learning_rate': 5.643738977072311e-08, 'epoch': 1.99}\n",
      "{'loss': 0.0001, 'grad_norm': 0.005415344191255889, 'learning_rate': 5.291005291005291e-08, 'epoch': 1.99}\n",
      "{'loss': 0.0, 'grad_norm': 0.0032148833654706685, 'learning_rate': 4.938271604938272e-08, 'epoch': 1.99}\n",
      "{'loss': 0.0, 'grad_norm': 1.4026965123685529e-05, 'learning_rate': 4.585537918871253e-08, 'epoch': 1.99}\n",
      "{'loss': 0.0, 'grad_norm': 0.0026195195869396066, 'learning_rate': 4.2328042328042335e-08, 'epoch': 1.99}\n",
      "{'loss': 0.0, 'grad_norm': 0.004420981716754515, 'learning_rate': 3.880070546737214e-08, 'epoch': 1.99}\n",
      "{'loss': 0.0, 'grad_norm': 0.00011741647931916769, 'learning_rate': 3.527336860670194e-08, 'epoch': 2.0}\n",
      "{'loss': 0.0, 'grad_norm': 0.003142961724610982, 'learning_rate': 3.174603174603175e-08, 'epoch': 2.0}\n",
      "{'loss': 0.0, 'grad_norm': 0.00018218223784848898, 'learning_rate': 2.8218694885361557e-08, 'epoch': 2.0}\n",
      "{'loss': 0.0003, 'grad_norm': 0.04631983050335103, 'learning_rate': 2.469135802469136e-08, 'epoch': 2.0}\n",
      "{'loss': 0.0, 'grad_norm': 5.1132348345306354e-05, 'learning_rate': 2.1164021164021167e-08, 'epoch': 2.0}\n",
      "{'loss': 0.0003, 'grad_norm': 0.03240276881060448, 'learning_rate': 1.763668430335097e-08, 'epoch': 2.0}\n",
      "{'loss': 0.0001, 'grad_norm': 0.011975699374629016, 'learning_rate': 1.4109347442680778e-08, 'epoch': 2.0}\n",
      "{'loss': 0.0967, 'grad_norm': 8.654642224335447, 'learning_rate': 1.0582010582010584e-08, 'epoch': 2.0}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 3150/3150 [13:10<00:00,  3.72it/s]12/23/2024 06:48:17 - INFO - FlagEmbedding.finetune.embedder.encoder_only.base.trainer -   Saving model checkpoint to ./test_encoder_only_base_bge-large-en-v1.5/checkpoint-3150\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'train_runtime': 799.0537, 'train_samples_per_second': 15.769, 'train_steps_per_second': 3.942, 'train_loss': 0.04348497095562163, 'epoch': 2.0}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 3150/3150 [13:19<00:00,  3.94it/s]\n",
      "12/23/2024 06:48:26 - INFO - FlagEmbedding.finetune.embedder.encoder_only.base.trainer -   Saving model checkpoint to ./test_encoder_only_base_bge-large-en-v1.5\n",
      "[rank0]:[W1223 06:48:28.948814944 ProcessGroupNCCL.cpp:1250] Warning: WARNING: process group has NOT been destroyed before we destruct ProcessGroupNCCL. On normal program exit, the application should call destroy_process_group to ensure that any pending NCCL operations have finished in this process. In rare cases this process can exit before this point and block the progress of another member of the process group. This constraint has always been present,  but this warning has only been added since PyTorch 2.4 (function operator())\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "torchrun --nproc_per_node 2 \\\n",
    "\t-m FlagEmbedding.finetune.embedder.encoder_only.base \\\n",
    "\t--model_name_or_path BAAI/bge-large-en-v1.5 \\\n",
    "    --cache_dir ./cache/model \\\n",
    "    --train_data ./ft_data/training.json \\\n",
    "    --cache_path ./cache/data \\\n",
    "    --train_group_size 8 \\\n",
    "    --query_max_len 512 \\\n",
    "    --passage_max_len 512 \\\n",
    "    --pad_to_multiple_of 8 \\\n",
    "    --query_instruction_for_retrieval 'Represent this sentence for searching relevant passages: ' \\\n",
    "    --query_instruction_format '{}{}' \\\n",
    "    --knowledge_distillation False \\\n",
    "\t--output_dir ./test_encoder_only_base_bge-large-en-v1.5 \\\n",
    "    --overwrite_output_dir \\\n",
    "    --learning_rate 1e-5 \\\n",
    "    --fp16 \\\n",
    "    --num_train_epochs 2 \\\n",
    "    --per_device_train_batch_size 2 \\\n",
    "    --dataloader_drop_last True \\\n",
    "    --warmup_ratio 0.1 \\\n",
    "    --gradient_checkpointing \\\n",
    "    --deepspeed config/ds_stage0.json \\\n",
    "    --logging_steps 1 \\\n",
    "    --save_steps 1000 \\\n",
    "    --negatives_cross_device \\\n",
    "    --temperature 0.02 \\\n",
    "    --sentence_pooling_method cls \\\n",
    "    --normalize_embeddings True \\\n",
    "    --kd_loss_type kl_div"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "ft",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}