diff --git a/notebooks/01-llama-server-completions-api-endpoints.ipynb b/notebooks/01-llama-server-completions-api-endpoints.ipynb new file mode 100644 index 0000000..2e0e1a5 --- /dev/null +++ b/notebooks/01-llama-server-completions-api-endpoints.ipynb @@ -0,0 +1,1075 @@ +{ + "cells": [ + { + "cell_type": "code", + "execution_count": 101, + "id": "422dc501-69bf-4105-b800-9f14c596c67d", + "metadata": {}, + "outputs": [], + "source": [ + "import json\n", + "import requests\n", + "from IPython.display import JSON, Markdown\n" + ] + }, + { + "cell_type": "markdown", + "id": "6ff78001-5c11-499c-9bb9-e150fa928ac9", + "metadata": {}, + "source": [ + "# Completion API Endpoints" + ] + }, + { + "cell_type": "markdown", + "id": "75e14ac0-87fd-4331-834b-8d191f768d67", + "metadata": {}, + "source": [ + "## Start `llama-server`\n", + "\n", + "Open a new terminal and run the following command to start a new server." + ] + }, + { + "cell_type": "markdown", + "id": "bf78f2fd-7acc-454c-8d94-b43d12b14cbe", + "metadata": {}, + "source": [ + "```bash\n", + "MODEL=\"./models/gemma-1.1-7b-it.Q4_K_M.gguf\"\n", + "llama-server \\\n", + " --model $MODEL \\\n", + " --host localhost \\\n", + " --port 8080\n", + "```" + ] + }, + { + "cell_type": "markdown", + "id": "1a5a775a-a31e-49d1-8c48-cfb4d911bc69", + "metadata": {}, + "source": [ + "## Basic usage" + ] + }, + { + "cell_type": "markdown", + "id": "89551ed3-eb58-44a3-96e1-09649ad68271", + "metadata": {}, + "source": [ + "\n", + " `prompt`: Provide the prompt for this completion as a string or as an array of strings or numbers representing tokens. Internally, if `cache_prompt` is `true`, the prompt is compared to the previous completion and only the \"unseen\" suffix is evaluated. A `BOS` token is inserted at the start, if all of the following conditions are true:\n", + "\n", + " - The prompt is a string or an array with the first element given as a string\n", + " - The model's `tokenizer.ggml.add_bos_token` metadata is `true`\n", + "\n", + " `cache_prompt`: Re-use KV cache from a previous request if possible. This way the common prefix does not have to be re-processed, only the suffix that differs between the requests. Because (depending on the backend) the logits are **not** guaranteed to be bit-for-bit identical for different batch sizes (prompt processing vs. token generation) enabling this option can cause nondeterministic results. Default: `false`\n", + "\n", + " `stream`: It allows receiving each predicted token in real-time instead of waiting for the completion to finish. To enable this, set to `true`.\n", + "\n", + " `stop`: Specify a JSON array of stopping strings.\n", + " These words will not be included in the completion, so make sure to add them to the prompt for the next iteration. Default: `[]`" + ] + }, + { + "cell_type": "code", + "execution_count": 110, + "id": "646df475-f0d6-4ff4-b3ea-82a000180472", + "metadata": {}, + "outputs": [], + "source": [ + "response = requests.post(\n", + " url=\"http://localhost:8080/completion\",\n", + " json={\n", + " \"prompt\": \"Why is the sky blue?\",\n", + " \"cache_prompt\": False,\n", + " \"stream\": False,\n", + " \"stop\": [],\n", + " }\n", + ")" + ] + }, + { + "cell_type": "code", + "execution_count": 113, + "id": "d97c3018-5a89-43b6-8d8f-1f979f4d4976", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n" + ] + } + ], + "source": [ + "print(response)" + ] + }, + { + "cell_type": "code", + "execution_count": 116, + "id": "b184d94c-2150-4674-acab-350bce169332", + "metadata": {}, + "outputs": [ + { + "data": { + "application/json": { + "content": "\n\n**Answer:**\n\nThe sky is blue due to a phenomenon called **Rayleigh scattering**. \n\n* Sunlight is composed of all the colors of the rainbow, each with a specific wavelength.\n* When sunlight interacts with molecules in the atmosphere, such as nitrogen and oxygen, the molecules scatter the light. \n* Different wavelengths of light are scattered differently. \n\n**Rayleigh scattering:**\n\n- Shorter wavelengths of light (like blue light) are scattered more efficiently than longer wavelengths. \n- Since blue light is scattered more, it reaches our eyes from all directions in the sky, making the sky appear blue.\n\n**Other factors influencing the color of the sky:**\n\n- Time of day (sun angle)\n- Latitude and altitude\n- Clouds and pollution\n- Sun's position in the sky", + "generation_settings": { + "dynatemp_exponent": 1, + "dynatemp_range": 0, + "frequency_penalty": 0, + "grammar": "", + "ignore_eos": false, + "max_tokens": -1, + "min_keep": 0, + "min_p": 0.05000000074505806, + "mirostat": 0, + "mirostat_eta": 0.10000000149011612, + "mirostat_tau": 5, + "model": "./models/gemma-1.1-7b-it.Q4_K_M.gguf", + "n_ctx": 8192, + "n_discard": 0, + "n_keep": 0, + "n_predict": -1, + "n_probs": 0, + "penalize_nl": false, + "presence_penalty": 0, + "repeat_last_n": 64, + "repeat_penalty": 1, + "samplers": [ + "top_k", + "tfs_z", + "typ_p", + "top_p", + "min_p", + "temperature" + ], + "seed": 4294967295, + "seed_cur": 3670435482, + "stop": [], + "stream": false, + "temperature": 0.800000011920929, + "tfs_z": 1, + "top_k": 40, + "top_p": 0.949999988079071, + "typical_p": 1 + }, + "id_slot": 0, + "index": 0, + "model": "./models/gemma-1.1-7b-it.Q4_K_M.gguf", + "prompt": "Why is the sky blue?", + "stop": true, + "stopped_eos": true, + "stopped_limit": false, + "stopped_word": false, + "stopping_word": "", + "timings": { + "predicted_ms": 10549.645, + "predicted_n": 166, + "predicted_per_second": 15.735126632223169, + "predicted_per_token_ms": 63.55207831325301, + "prompt_ms": 420.141, + "prompt_n": 7, + "prompt_per_second": 16.661073306342395, + "prompt_per_token_ms": 60.02014285714286 + }, + "tokens_cached": 172, + "tokens_evaluated": 7, + "tokens_predicted": 166, + "truncated": false + }, + "text/plain": [ + "" + ] + }, + "execution_count": 116, + "metadata": { + "application/json": { + "expanded": false, + "root": "root" + } + }, + "output_type": "execute_result" + } + ], + "source": [ + "json_data = json.loads(response.content)\n", + "JSON(json_data)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "1b648612-3dbf-4de1-9ba2-adccd55c229f", + "metadata": {}, + "outputs": [], + "source": [ + "_generated_text = json_data[\"content\"]\n", + "Markdown(_generated_text)" + ] + }, + { + "cell_type": "markdown", + "id": "c21ef3c4-32eb-4850-9ec4-8e1be9a7ea37", + "metadata": {}, + "source": [ + "## Options" + ] + }, + { + "cell_type": "markdown", + "id": "47d601c7-051e-4ad7-9640-3c299affc358", + "metadata": {}, + "source": [ + "### Random Number Generator (RNG) Seed\n", + "\n", + "The RNG seed is used to initialize the random number generator that influences the text generation process. By setting a specific value for `seed` you can obtain consistent and reproducible results across multiple runs with the same input and settings. This can be helpful for testing, debugging, or comparing the effects of different options on the generated text to see when they diverge. If the seed is set to a value less than 0, a random seed will be used, which will result in different outputs on each run. The default value is -1 which will choose a random value for `seed`." + ] + }, + { + "cell_type": "code", + "execution_count": 128, + "id": "2bd3b1f7-fe18-4c13-a985-8c2c60128c4b", + "metadata": {}, + "outputs": [], + "source": [ + "response = requests.post(\n", + " url=\"http://localhost:8080/completion\",\n", + " json={\n", + " \"prompt\": \"Why is the sky blue?\",\n", + " \"cache_prompt\": False,\n", + " \"stream\": False,\n", + " \"stop\": [],\n", + " \"seed\": 42\n", + " }\n", + ")" + ] + }, + { + "cell_type": "code", + "execution_count": 129, + "id": "95555aea-c700-41ab-aadf-dd272c7a2fea", + "metadata": {}, + "outputs": [ + { + "data": { + "text/markdown": [ + "\n", + "\n", + "**Answer:**\n", + "\n", + "The sky is blue due to a phenomenon called **Rayleigh scattering**. \n", + "\n", + "* Sunlight is composed of all the colors of the rainbow, each with a specific wavelength. \n", + "* When sunlight interacts with molecules in the atmosphere, such as nitrogen and oxygen, the different wavelengths are scattered in different directions. \n", + "* Shorter wavelengths of light, like blue light, are scattered more efficiently than longer wavelengths. \n", + "* Since our eyes are most sensitive to blue light, we perceive the sky as blue." + ], + "text/plain": [ + "" + ] + }, + "execution_count": 129, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "_json_data = json.loads(response.content)\n", + "_generated_text = _json_data[\"content\"]\n", + "Markdown(_generated_text)" + ] + }, + { + "cell_type": "markdown", + "id": "a722becf-8446-40f4-826b-da19189b45d1", + "metadata": {}, + "source": [ + "### Number of Tokens to Predict\n", + "\n", + "The `n_predict` (default: -1) controls the number of tokens the model generates in response to the input prompt. By adjusting this value, you can influence the length of the generated text. A higher value will result in longer text, while a lower value will produce shorter text.\n", + "\n", + "Even though all models have a finite context window, a value of -1 will enable *infinite* text generation. How? When the context window is full half of the tokens after `n_keep` will be discarded. The context must then be re-evaluated before generation can resume. On large models and/or large context windows, this can result in a significant pause in output. If the output delay is undesirable, a value of -2 will stop generation immediately when the context is filled.\n", + "\n", + "It is important to note that the generated text may be shorter than the specified number of tokens if an End-of-Sequence (EOS) token or a reverse prompt is encountered. In interactive mode, text generation will pause and control will be returned to the user. In non-interactive mode, the program will end. In both cases, the text generation may stop before reaching the specified `n_predict` value. If you want the model to keep going without ever producing End-of-Sequence on its own, you can use the `ignore_eos` parameter.\n", + "\n", + "\n", + " `n_keep`: Specify the number of tokens from the prompt to retain when the context size is exceeded and tokens need to be discarded. The number excludes the BOS token.\n", + " By default, this value is set to `0`, meaning no tokens are kept. Use `-1` to retain all tokens from the prompt.\n", + "\n", + " `min_keep`: If greater than 0, force samplers to return N possible tokens at minimum. Default: `0`\n", + "\n", + " `t_max_predict_ms`: Set a time limit in milliseconds for the prediction (a.k.a. text-generation) phase. The timeout will trigger if the generation takes more than the specified time (measured since the first token was generated) and if a new-line character has already been generated. Useful for FIM applications. Default: `0`, which is disabled." + ] + }, + { + "cell_type": "code", + "execution_count": 130, + "id": "d487725b-45f9-41ec-a483-22d85f3b9d8d", + "metadata": {}, + "outputs": [], + "source": [ + "response = requests.post(\n", + " url=\"http://localhost:8080/completion\",\n", + " json={\n", + " \"prompt\": \"Why is the sky blue?\",\n", + " \"cache_prompt\": False,\n", + " \"stream\": False,\n", + " \"stop\": [],\n", + " \"seed\": 42,\n", + " \"n_predict\": -1,\n", + " \"n_keep\": 0,\n", + " \"min_keep\": 0,\n", + " \"ignore_eos\": False\n", + " }\n", + ")" + ] + }, + { + "cell_type": "code", + "execution_count": 131, + "id": "4dcd35c8-d39d-4bca-8705-1bd10a302a39", + "metadata": {}, + "outputs": [ + { + "data": { + "text/markdown": [ + "\n", + "\n", + "**Answer:**\n", + "\n", + "The sky is blue due to a phenomenon called **Rayleigh scattering**. \n", + "\n", + "* Sunlight is composed of all the colors of the rainbow, each with a specific wavelength. \n", + "* When sunlight interacts with molecules in the atmosphere, such as nitrogen and oxygen, the different wavelengths are scattered in different directions. \n", + "* Shorter wavelengths of light, like blue light, are scattered more efficiently than longer wavelengths. \n", + "* Since our eyes are most sensitive to blue light, we perceive the sky as blue." + ], + "text/plain": [ + "" + ] + }, + "execution_count": 131, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "_json_data = json.loads(response.content)\n", + "_generated_text = _json_data[\"content\"]\n", + "Markdown(_generated_text)" + ] + }, + { + "cell_type": "markdown", + "id": "973d8a35-d587-4c99-b3fb-5307c2bcb50a", + "metadata": {}, + "source": [ + "### Temperature\n", + "\n", + "The `temperature` hyperparameter that controls the randomness of the generated text. It affects the probability distribution of the model's output tokens. A higher temperature makes the output more random and creative, while a lower temperature makes the output more focused, deterministic, and conservative.\n", + "\n", + " `dynatemp_range`: Dynamic temperature range. The final temperature will be in the range of `[temperature - dynatemp_range; temperature + dynatemp_range]` Default: `0.0`, which is disabled.\n", + "\n", + " `dynatemp_exponent`: Dynamic temperature exponent. Default: `1.0`" + ] + }, + { + "cell_type": "code", + "execution_count": 132, + "id": "90298924-b5ef-417a-8a83-7c62cdd79fef", + "metadata": {}, + "outputs": [], + "source": [ + "response = requests.post(\n", + " url=\"http://localhost:8080/completion\",\n", + " json={\n", + " \"prompt\": \"Why is the sky blue?\",\n", + " \"cache_prompt\": False,\n", + " \"stream\": False,\n", + " \"stop\": [],\n", + " \"seed\": 42,\n", + " \"n_predict\": -1,\n", + " \"n_keep\": 0,\n", + " \"min_keep\": 0,\n", + " \"ignore_eos\": False,\n", + " \"temperature\": 0.8,\n", + " \"dynatemp_range\": 0.1,\n", + " \"dynatemp_exponent\": 1.0\n", + " }\n", + ")" + ] + }, + { + "cell_type": "code", + "execution_count": 133, + "id": "56204f43-6125-495d-a0ca-0b771a6c4130", + "metadata": {}, + "outputs": [ + { + "data": { + "text/markdown": [ + "\n", + "\n", + "**Answer:**\n", + "\n", + "The sky is blue due to a phenomenon called **Rayleigh scattering**. \n", + "\n", + "* Sunlight is composed of all the colors of the rainbow, each with a specific wavelength. \n", + "* When sunlight interacts with molecules in the atmosphere, such as nitrogen and oxygen, the different wavelengths are scattered in different directions. \n", + "* Shorter wavelengths of light, such as blue light, are scattered more efficiently than longer wavelengths. \n", + "* As a result, more blue light is scattered in all directions, reaching our eyes and making the sky appear blue." + ], + "text/plain": [ + "" + ] + }, + "execution_count": 133, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "_json_data = json.loads(response.content)\n", + "_generated_text = _json_data[\"content\"]\n", + "\n", + "Markdown(_generated_text)" + ] + }, + { + "cell_type": "markdown", + "id": "8c773e60-24a4-4a36-8855-9cda246739b3", + "metadata": {}, + "source": [ + "#### 1.1.6 Repeat Penalty\n", + "\n", + "The `repeat_penalty` option helps prevent the model from generating repetitive or monotonous text. A higher value (e.g., 1.5) will penalize repetitions more strongly, while a lower value (e.g., 0.9) will be more lenient. The default value is 1.1.\n", + "\n", + "The `repeat_last_n` option controls the number of tokens in the history to consider for penalizing repetition. A larger value will look further back in the generated text to prevent repetitions, while a smaller value will only consider recent tokens. A value of 0 disables the penalty, and a value of -1 sets the number of tokens considered equal to the context size, `ctx_size`. The default value is 64. \n", + "\n", + " `penalize_nl`: Penalize newline tokens when applying the repeat penalty. Default: `true`\n" + ] + }, + { + "cell_type": "code", + "execution_count": 134, + "id": "941a7482-bc32-4cff-ac08-8489c45dea0d", + "metadata": {}, + "outputs": [], + "source": [ + "response = requests.post(\n", + " url=\"http://localhost:8080/completion\",\n", + " json={\n", + " \"prompt\": \"Why is the sky blue?\",\n", + " \"cache_prompt\": False,\n", + " \"stream\": False,\n", + " \"stop\": [],\n", + " \"seed\": 42,\n", + " \"n_predict\": -1,\n", + " \"n_keep\": 0,\n", + " \"min_keep\": 0,\n", + " \"ignore_eos\": False,\n", + " \"temperature\": 0.8,\n", + " \"dynatemp_range\": 0.1,\n", + " \"dynatemp_exponent\": 1.0,\n", + " \"repeat_penalty\": 1.1,\n", + " \"repeat_last_n\": 64,\n", + " \"penalize_nl\": True\n", + " }\n", + ")" + ] + }, + { + "cell_type": "code", + "execution_count": 135, + "id": "795b009c-3f02-4183-a0eb-34b8424bfeaf", + "metadata": {}, + "outputs": [ + { + "data": { + "text/markdown": [ + "\n", + "\n", + "**Answer:** \n", + "The sky is blue due to the process of **Rayleigh scattering**. Sunlight consists mainly of all wavelengths in a range from violet through red. When sunlight interacts with molecules like nitrogen and oxygen, shorter wavelength light (violet &blue) gets scattered more efficiently than longer wave length lights(orange&red).\n", + "\n", + "* Shorter waves have higher frequency/energy per unit area which results into greater scattering power against the particles of air .\n", + "**The blue wavelengths are dispersed in all directions by these gases.** \n", + "\n", + "\n", + "- Most of this diffused light reaches our eyes from a relatively small portion (about one degree) directly overhead. That's why we see mainly skyblue during clear weather when there isn’t much dust or cloud to absorb the scattered sunlight .\n", + "\n", + "**Factors influencing colour of Sky:**\n", + "* Time & Latitude - different time zones experience differently coloured skies due variations in sun elevation and composition of atmosphere \n", + "\n", + "\n", + "- Cloud Coverage/ Dust particles – clouds block direct contact between Sunlight& air molecules thereby limiting scattering. Similarly, large dust particle can scatter all wavelengths equally leading to a less blue sky ." + ], + "text/plain": [ + "" + ] + }, + "execution_count": 135, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "_json_data = json.loads(response.content)\n", + "_generated_text = _json_data[\"content\"]\n", + "\n", + "Markdown(_generated_text)" + ] + }, + { + "cell_type": "markdown", + "id": "1e2f173c-2dcc-4446-a3cb-c3c3c875bb30", + "metadata": {}, + "source": [ + "### Top-K Sampling\n", + "\n", + "Top-k sampling is a text generation method that selects the next token only from the `--top-k` most likely tokens predicted by the model. It helps reduce the risk of generating low-probability or nonsensical tokens, but it may also limit the diversity of the output. A higher value for top-k (e.g., 100) will consider more tokens and lead to more diverse text, while a lower value (e.g., 10) will focus on the most probable tokens and generate more conservative text. The default value is 40.\n", + "\n", + " `top_k`: Limit the next token selection to the K most probable tokens. Default: `40`\n" + ] + }, + { + "cell_type": "code", + "execution_count": 136, + "id": "608fd603-9339-452a-8dae-a4c14c7d7da2", + "metadata": {}, + "outputs": [], + "source": [ + "response = requests.post(\n", + " url=\"http://localhost:8080/completion\",\n", + " json={\n", + " \"prompt\": \"Why is the sky blue?\",\n", + " \"cache_prompt\": False,\n", + " \"stream\": False,\n", + " \"stop\": [],\n", + " \"seed\": 42,\n", + " \"n_predict\": -1,\n", + " \"n_keep\": 0,\n", + " \"min_keep\": 0,\n", + " \"ignore_eos\": False,\n", + " \"temperature\": 0.8,\n", + " \"dynatemp_range\": 0.1,\n", + " \"dynatemp_exponent\": 1.0,\n", + " \"repeat_penalty\": 1.1,\n", + " \"repeat_last_n\": 64,\n", + " \"penalize_nl\": True,\n", + " \"top_k\": 40,\n", + " }\n", + ")" + ] + }, + { + "cell_type": "code", + "execution_count": 137, + "id": "9de0ee1b-9085-42eb-9fab-ed365e810129", + "metadata": {}, + "outputs": [ + { + "data": { + "text/markdown": [ + "\n", + "\n", + "**Answer:** \n", + "The sky is blue due to the process of **Rayleigh scattering**. Sunlight consists mainly of all wavelengths in a range from violet through red. When sunlight interacts with molecules like nitrogen and oxygen, shorter wavelength light (violet &blue) gets scattered more efficiently than longer wave length lights(orange&red).\n", + "\n", + "* Shorter waves have higher frequency/energy per unit area which results into greater scattering power against the particles of air .\n", + "**The blue wavelengths are dispersed in all directions by these gases.** \n", + "\n", + "\n", + "- Most of this diffused light reaches our eyes from a relatively small portion (about one degree) directly overhead. That's why we see mainly skyblue during clear weather when there isn’t much dust or cloud to absorb the scattered sunlight .\n", + "\n", + "**Factors influencing colour of Sky:**\n", + "* Time & Latitude - different time zones experience differently coloured skies due variations in sun elevation and composition of atmosphere \n", + "\n", + "\n", + "- Cloud Coverage/ Dust particles – clouds block direct contact between Sunlight& air molecules thereby limiting scattering. Similarly, large dust particle can scatter all wavelengths equally leading to a less blue sky ." + ], + "text/plain": [ + "" + ] + }, + "execution_count": 137, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "json_data = json.loads(response.content)\n", + "_generated_text = json_data[\"content\"]\n", + "\n", + "Markdown(_generated_text)" + ] + }, + { + "cell_type": "markdown", + "id": "3795881f-dcf1-4fdd-8295-4a53d12078d4", + "metadata": {}, + "source": [ + "### Top-P Sampling\n", + "\n", + "Top-p sampling, `top-p`, also known as nucleus sampling, is another text generation method that selects the next token from a subset of tokens that together have a cumulative probability of at least p. This method provides a balance between diversity and quality by considering both the probabilities of tokens and the number of tokens to sample from. A higher value for top-p (e.g., 0.95) will lead to more diverse text, while a lower value (e.g., 0.5) will generate more focused and conservative text. The default value is 0.9.\n", + "\n", + " `top_p`: Limit the next token selection to a subset of tokens with a cumulative probability above a threshold P. Default: `0.95`" + ] + }, + { + "cell_type": "code", + "execution_count": 138, + "id": "8b882d6e-ed33-45c8-8ec8-ca3a2a767395", + "metadata": {}, + "outputs": [], + "source": [ + "response = requests.post(\n", + " url=\"http://localhost:8080/completion\",\n", + " json={\n", + " \"prompt\": \"Why is the sky blue?\",\n", + " \"cache_prompt\": False,\n", + " \"stream\": False,\n", + " \"stop\": [],\n", + " \"seed\": 42,\n", + " \"n_predict\": -1,\n", + " \"n_keep\": 0,\n", + " \"min_keep\": 0,\n", + " \"ignore_eos\": False,\n", + " \"temperature\": 0.8,\n", + " \"dynatemp_range\": 0.1,\n", + " \"dynatemp_exponent\": 1.0,\n", + " \"repeat_penalty\": 1.1,\n", + " \"repeat_last_n\": 64,\n", + " \"penalize_nl\": True,\n", + " \"top_k\": 40,\n", + " \"top_p\": 0.95,\n", + " }\n", + ")" + ] + }, + { + "cell_type": "code", + "execution_count": 139, + "id": "0dc9538a-9d32-407c-9bca-6f568643dc33", + "metadata": {}, + "outputs": [ + { + "data": { + "text/markdown": [ + "\n", + "\n", + "**Answer:** \n", + "The sky is blue due to the process of **Rayleigh scattering**. Sunlight consists mainly of all wavelengths in a range from violet through red. When sunlight interacts with molecules like nitrogen and oxygen, shorter wavelength light (violet &blue) gets scattered more efficiently than longer wave length lights(orange&red).\n", + "\n", + "* Shorter waves have higher frequency/energy per unit area which results into greater scattering power against the particles of air .\n", + "**The blue wavelengths are dispersed in all directions by these gases.** \n", + "\n", + "\n", + "- Most of this diffused light reaches our eyes from a relatively small portion (about one degree) directly overhead. That's why we see mainly skyblue during clear weather when there isn’t much dust or cloud to absorb the scattered sunlight .\n", + "\n", + "**Factors influencing colour of Sky:**\n", + "* Time & Latitude - different time zones experience differently coloured skies due variations in sun elevation and composition of atmosphere \n", + "\n", + "\n", + "- Cloud Coverage/ Dust particles – clouds block direct contact between Sunlight& air molecules thereby limiting scattering. Similarly, large dust particle can scatter all wavelengths equally leading to a less blue sky ." + ], + "text/plain": [ + "" + ] + }, + "execution_count": 139, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "json_data = json.loads(response.content)\n", + "_generated_text = json_data[\"content\"]\n", + "\n", + "Markdown(_generated_text)" + ] + }, + { + "cell_type": "markdown", + "id": "df20935d-0211-44a7-91f0-8df8953d8779", + "metadata": {}, + "source": [ + "### Min-P Sampling\n", + "\n", + "The `--min-p` sampling method sets a minimum base probability threshold for token selection and aims to ensure a balance of quality and variety in the generated text. The `--min-p` method was designed as an alternative to `--top-p`. The parameter $p$ represents the minimum probability for a token to be considered, relative to the probability of the most likely token. For example, with $p=0.05$ and the most likely token having a probability of 0.9, logits with a value less than 0.045 are filtered out. The default value is 0.1.\n", + "\n", + " `min_p`: The minimum probability for a token to be considered, relative to the probability of the most likely token. Default: `0.05`\n", + " " + ] + }, + { + "cell_type": "code", + "execution_count": 140, + "id": "3480cb23-7a09-4b60-9020-165231f72d9e", + "metadata": {}, + "outputs": [], + "source": [ + "response = requests.post(\n", + " url=\"http://localhost:8080/completion\",\n", + " json={\n", + " \"prompt\": \"Why is the sky blue?\",\n", + " \"cache_prompt\": False,\n", + " \"stream\": False,\n", + " \"stop\": [],\n", + " \"seed\": 42,\n", + " \"n_predict\": -1,\n", + " \"n_keep\": 0,\n", + " \"min_keep\": 0,\n", + " \"ignore_eos\": False,\n", + " \"temperature\": 0.8,\n", + " \"dynatemp_range\": 0.1,\n", + " \"dynatemp_exponent\": 1.0,\n", + " \"repeat_penalty\": 1.1,\n", + " \"repeat_last_n\": 64,\n", + " \"penalize_nl\": True,\n", + " \"top_k\": 40,\n", + " \"top_p\": 0.95,\n", + " \"min_p\": 0.05,\n", + " }\n", + ")" + ] + }, + { + "cell_type": "code", + "execution_count": 141, + "id": "acace774-f746-4142-90c3-6a8c39196577", + "metadata": {}, + "outputs": [ + { + "data": { + "text/markdown": [ + "\n", + "\n", + "**Answer:** \n", + "The sky is blue due to the process of **Rayleigh scattering**. Sunlight consists mainly of all wavelengths in a range from violet through red. When sunlight interacts with molecules like nitrogen and oxygen, shorter wavelength light (violet &blue) gets scattered more efficiently than longer wave length lights(orange&red).\n", + "\n", + "* Shorter waves have higher frequency/energy per unit area which results into greater scattering power against the particles of air .\n", + "**The blue wavelengths are dispersed in all directions by these gases.** \n", + "\n", + "\n", + "- Most of this diffused light reaches our eyes from a relatively small portion (about one degree) directly overhead. That's why we see mainly skyblue during clear weather when there isn’t much dust or cloud to absorb the scattered sunlight .\n", + "\n", + "**Factors influencing colour of Sky:**\n", + "* Time & Latitude - different time zones experience differently coloured skies due variations in sun elevation and composition of atmosphere \n", + "\n", + "\n", + "- Cloud Coverage/ Dust particles – clouds block direct contact between Sunlight& air molecules thereby limiting scattering. Similarly, large dust particle can scatter all wavelengths equally leading to a less blue sky ." + ], + "text/plain": [ + "" + ] + }, + "execution_count": 141, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "json_data = json.loads(response.content)\n", + "_generated_text = json_data[\"content\"]\n", + "\n", + "Markdown(_generated_text)" + ] + }, + { + "cell_type": "markdown", + "id": "2cda8568-978c-4f49-8473-420182261a00", + "metadata": {}, + "source": [ + "### Locally Typical Sampling\n", + "\n", + "Locally typical sampling, `typical_p`, promotes the generation of contextually coherent and diverse text by sampling tokens that are typical or expected based on the surrounding context. By setting the parameter $p$ between 0 and 1, you can control the balance between producing text that is locally coherent and diverse. The default setting is $p=1.0$, which disables locally typical sampling.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 142, + "id": "5b8153e3-e2ca-4cb2-8fa5-cf897e0f4029", + "metadata": {}, + "outputs": [], + "source": [ + "response = requests.post(\n", + " url=\"http://localhost:8080/completion\",\n", + " json={\n", + " \"prompt\": \"Why is the sky blue?\",\n", + " \"cache_prompt\": False,\n", + " \"stream\": False,\n", + " \"stop\": [],\n", + " \"seed\": 42,\n", + " \"n_predict\": -1,\n", + " \"n_keep\": 0,\n", + " \"min_keep\": 0,\n", + " \"ignore_eos\": False,\n", + " \"temperature\": 0.8,\n", + " \"dynatemp_range\": 0.0,\n", + " \"dynatemp_exponent\": 1.0,\n", + " \"repeat_penalty\": 1.1,\n", + " \"repeat_last_n\": 64,\n", + " \"penalize_nl\": True,\n", + " \"top_k\": 40,\n", + " \"top_p\": 0.95,\n", + " \"min_p\": 0.05,\n", + " \"typical_p\": 1.0,\n", + " }\n", + ")" + ] + }, + { + "cell_type": "code", + "execution_count": 143, + "id": "fe297741-2447-44f1-883e-d683d87c68c3", + "metadata": {}, + "outputs": [ + { + "data": { + "text/markdown": [ + "\n", + "\n", + "**Answer:** \n", + "The sky is blue due to the process of **Rayleigh scattering**. Sunlight consists mainly of all wavelengths or colors. When sunlight interacts with molecules in Earth's atmosphere, like nitrogen and oxygen atoms these particles scatter light rays randomly without altering their direction significantly . But different color lights are scattered differently based on wavelength:\n", + "\n", + "- Shorter Wavelength (blue/violet) - gets dispersed more efficiently\n", + "_LongerWavelength_(red / infra red)_ is less effectively diffused. \n", + "\n", + "\n", + "**The blue wavelengths of sunlight:** travel in straight lines until they reach our eyes, giving us the impression that sky appears **BLUE**." + ], + "text/plain": [ + "" + ] + }, + "execution_count": 143, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "json_data = json.loads(response.content)\n", + "_generated_text = json_data[\"content\"]\n", + "\n", + "Markdown(_generated_text)" + ] + }, + { + "cell_type": "markdown", + "id": "74a53574-f7ca-4243-ace6-8b3a11928337", + "metadata": {}, + "source": [ + "### Mirostat Sampling\n", + "\n", + "Mirostat is an algorithm that actively maintains the quality of generated text within a desired range during text generation. It aims to strike a balance between coherence and diversity, avoiding low-quality output caused by excessive repetition (boredom traps) or incoherence (confusion traps). To enable Mirostat sampling set `mirostat` to 1 = Mirostat 1.0 or 2 = Mirostat 2.0. By default Mirostat sampling is disabled `mirostat` to 0.\n", + "\n", + "The `mirostat_lr` option sets the Mirostat learning rate (eta). The learning rate influences how quickly the algorithm responds to feedback from the generated text. A lower learning rate will result in slower adjustments, while a higher learning rate will make the algorithm more responsive. The default value is `0.1`.\n", + "\n", + "The `mirostat_tau` option sets the Mirostat target entropy (tau), which represents the desired perplexity value for the generated text. Adjusting the target entropy allows you to control the balance between coherence and diversity in the generated text. A lower value will result in more focused and coherent text, while a higher value will lead to more diverse and potentially less coherent text. The default value is `5.0`.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 125, + "id": "709c9414-9418-4d64-9041-c0fbe63ddec6", + "metadata": {}, + "outputs": [], + "source": [ + "response = requests.post(\n", + " url=\"http://localhost:8080/completion\",\n", + " json={\n", + " \"prompt\": \"Why is the sky blue?\",\n", + " \"cache_prompt\": False,\n", + " \"stream\": False,\n", + " \"stop\": [],\n", + " \"seed\": 42,\n", + " \"n_predict\": -1,\n", + " \"n_keep\": 0,\n", + " \"min_keep\": 0,\n", + " \"ignore_eos\": False,\n", + " \"temperature\": 0.8,\n", + " \"dynatemp_range\": 0.0,\n", + " \"dynatemp_exponent\": 1.0,\n", + " \"repeat_penalty\": 1.1,\n", + " \"repeat_last_n\": 64,\n", + " \"penalize_nl\": True,\n", + " \"top_k\": 40,\n", + " \"top_p\": 0.95,\n", + " \"min_p\": 0.05,\n", + " \"typical_p\": 1.0,\n", + " \"mirostat\": 0.0,\n", + " \"mirostat_tau\": 5.0,\n", + " \"mirostat_eta\": 0.1 \n", + " \n", + " }\n", + ")" + ] + }, + { + "cell_type": "code", + "execution_count": 127, + "id": "97defc48-fc92-4768-a7de-82702bed5063", + "metadata": {}, + "outputs": [ + { + "data": { + "text/markdown": [ + "\n", + "\n", + "**Answer:**\n", + "\n", + "The sky is blue due to a phenomenon called **Rayleigh scattering**. \n", + "\n", + "* Sunlight is composed of all the colors of the rainbow, each with a specific wavelength.\n", + "* When sunlight interacts with molecules in the atmosphere, such as nitrogen and oxygen, the molecules scatter the light.\n", + "* Different wavelengths of light scatter differently. \n", + "* Shorter wavelengths of light, such as blue light, scatter more efficiently than longer wavelengths.\n", + "* Since blue light is scattered more evenly across the sky, we see a predominance of blue light when looking up at the sky.\n", + "\n", + "**Additional factors influencing the color of the sky:**\n", + "\n", + "* **Time of day:** The sun is higher in the sky during the day, resulting in more direct sunlight and a brighter blue sky.\n", + "* **Altitude:** Higher altitudes have thinner atmospheres, leading to less scattering and a clearer blue sky.\n", + "* **Cloud coverage:** Clouds block the sunlight and scatter less light, resulting in a less blue sky.\n", + "* **Pollution:** Pollution in the atmosphere can scatter different wavelengths of light, affecting the color of the sky." + ], + "text/plain": [ + "" + ] + }, + "execution_count": 127, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "json_data = json.loads(response.content)\n", + "_generated_text = json_data[\"content\"]\n", + "\n", + "Markdown(_generated_text)" + ] + }, + { + "cell_type": "markdown", + "id": "782c15cb-82b0-490e-a068-bf1690b52472", + "metadata": {}, + "source": [ + "### Samplers\n", + "\n", + "`samplers`: The order the samplers should be applied in. An array of strings representing sampler type names. If a sampler is not set, it will not be used. If a sampler is specified more than once, it will be applied multiple times. Default: `[\"top_k\", \"tfs_z\", \"typical_p\", \"top_p\", \"min_p\", \"temperature\"]` - these are all the available values." + ] + }, + { + "cell_type": "code", + "execution_count": 144, + "id": "f2b135e2-33c3-4178-8346-61445ef7bcca", + "metadata": {}, + "outputs": [], + "source": [ + "response = requests.post(\n", + " url=\"http://localhost:8080/completion\",\n", + " json={\n", + " \"prompt\": \"Why is the sky blue?\",\n", + " \"cache_prompt\": False,\n", + " \"stream\": False,\n", + " \"stop\": [],\n", + " \"seed\": 42,\n", + " \"n_predict\": -1,\n", + " \"n_keep\": 0,\n", + " \"min_keep\": 0,\n", + " \"ignore_eos\": False,\n", + " \"temperature\": 0.8,\n", + " \"dynatemp_range\": 0.0,\n", + " \"dynatemp_exponent\": 1.0,\n", + " \"repeat_penalty\": 1.1,\n", + " \"repeat_last_n\": 64,\n", + " \"penalize_nl\": True,\n", + " \"top_k\": 40,\n", + " \"top_p\": 0.95,\n", + " \"min_p\": 0.05,\n", + " \"typical_p\": 1.0,\n", + " \"mirostat\": 0.0,\n", + " \"mirostat_tau\": 5.0,\n", + " \"mirostat_eta\": 0.1,\n", + " \"samplers\": [\n", + " \"top_k\",\n", + " \"tfs_z\",\n", + " \"typical_p\",\n", + " \"top_p\",\n", + " \"min_p\",\n", + " \"temperature\",\n", + " ]\n", + " }\n", + ")" + ] + }, + { + "cell_type": "code", + "execution_count": 145, + "id": "bf2dbbd0-24f2-4d43-a5c9-a8cbae970d7c", + "metadata": {}, + "outputs": [ + { + "data": { + "text/markdown": [ + "\n", + "\n", + "**Answer:** \n", + "The sky is blue due to the process of **Rayleigh scattering**. Sunlight consists mainly of all wavelengths or colors. When sunlight interacts with molecules in Earth's atmosphere, like nitrogen and oxygen atoms these particles scatter light rays randomly without altering their direction significantly . But different color lights are scattered differently based on wavelength:\n", + "\n", + "- Shorter Wavelength (blue/violet) - gets dispersed more efficiently\n", + "_LongerWavelength_(red / infra red)_ is less effectively diffused. \n", + "\n", + "\n", + "**The blue wavelengths of sunlight:** travel in straight lines until they reach our eyes, giving us the impression that sky appears **BLUE**." + ], + "text/plain": [ + "" + ] + }, + "execution_count": 145, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "json_data = json.loads(response.content)\n", + "_generated_text = json_data[\"content\"]\n", + "\n", + "Markdown(_generated_text)" + ] + }, + { + "cell_type": "markdown", + "id": "15cec4a1-70d0-42f6-a4de-49fdbdc5dce3", + "metadata": {}, + "source": [] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.12.6" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +}