--- title: "Building a Code Generative AI Model" lang: "en" author: "Mohammad Abu Mattar" canonical: https://mkabumattar.com/post/building-a-code-generative-ai-model --- ![Blog post image for Building a Code Generative AI Model - How to build a Code Generative AI model as a software engineer: how AI writes code, step-by-step instructions to build your own, and answers to common questions about AI-generated code.](/_astro/hero.8WRw1hB0_21m6ix.webp) [Home](/)›[Blog](/blog)›[All Categories](/blog/categories)›[Artificial Intelligence](/blog/categories/artificial-intelligence) Blog [Prev in Artificial IntelligenceHow AI and LLMs Are Changing DevOps Incident Response](/blog/post/ai-powered-devops-incident-response-llms)[Next in Artificial IntelligenceGraphRAG Explained: Building Knowledge-Grounded LLM Systems](/blog/post/graphrag-explained-building-knowledge-grounded-llm-systems) [Artificial Intelligence](/blog/categories/artificial-intelligence)[Machine Learning](/blog/categories/machine-learning)[Software Engineering](/blog/categories/software-engineering)[Code Generation](/blog/categories/code-generation)[Python](/blog/categories/python) # Building a Code Generative AI Model [Mohammad Abu Mattar](/authors/mohammad-abu-mattar)Published: 29 Aug 202304 Mins read07 Mins listen [Markdown for AI(opens in a new tab)](/post/building-a-code-generative-ai-model/index.md "Open the plain-Markdown version of this page, for pasting into an AI tool") TL;DR How to build a Code Generative AI model as a software engineer: how AI writes code, step-by-step instructions to build your own, and answers to common questions about AI-generated code. Series [AI & LLM Engineering](/series/ai--llm-engineering)2/5 [PreviousUnderstanding Generative AI in Depth](/blog/post/understanding-generative-ai-in-depth)[NextAI is Not Real: A Software Engineering Perspective](/blog/post/ai-is-not-real) All posts in this series (5) Blog5 1. [Understanding Generative AI in Depth](/blog/post/understanding-generative-ai-in-depth) 2. [Building a Code Generative AI ModelYou are here](/blog/post/building-a-code-generative-ai-model) 3. [AI is Not Real: A Software Engineering Perspective](/blog/post/ai-is-not-real) 4. [Microsoft's Prompt Orchestration Markup Language (POML): Structuring the Future of AI Interaction](/blog/post/microsoft-poml-orchestrating-ai-prompts-for-llms) 5. [GraphRAG Explained: Building Knowledge-Grounded LLM Systems](/blog/post/graphrag-explained-building-knowledge-grounded-llm-systems) ### Building a Code Generative AI Model Contents [Introduction](#introduction)[Can Generative AI write code?](#can-generative-ai-write-code)[What is Generative AI computer code?](#what-is-generative-ai-computer-code)[How do I create a Generative AI for code?](#how-do-i-create-a-generative-ai-for-code)[Step 1: Environment setup](#step-1-environment-setup)[Step 2: Loading your AI model](#step-2-loading-your-ai-model)[Step 3: Generating code with your AI](#step-3-generating-code-with-your-ai)[Step 4: Running your Code Generative AI](#step-4-running-your-code-generative-ai)[Frequently asked questions](#frequently-asked-questions)[Q1: How does Generative AI understand programming languages?](#q1-how-does-generative-ai-understand-programming-languages)[Q2: Can Generative AI replace human programmers?](#q2-can-generative-ai-replace-human-programmers)[Q3: Are there ethical concerns related to AI-generated code?](#q3-are-there-ethical-concerns-related-to-ai-generated-code)[Q4: What are some practical applications of Code Generative AI?](#q4-what-are-some-practical-applications-of-code-generative-ai)[Q5: How can I fine-tune my Code Generative AI model?](#q5-how-can-i-fine-tune-my-code-generative-ai-model)[Conclusion](#conclusion)[References](#references) ## [Introduction](#introduction) Automation is a core part of software engineering. Code Generative AI now makes it possible to have an AI write code for you. In this article, we’ll build a Code Generative AI model from scratch and answer some common questions along the way. ## [Can Generative AI write code?](#can-generative-ai-write-code) Start with the obvious question: can Generative AI genuinely write code? Yes. Generative AI models, particularly those built on neural networks, have shown strong results generating human-like text, including code. These models train on large datasets covering various programming languages, which lets them produce code snippets that are both syntactically accurate and semantically meaningful. ## [What is Generative AI computer code?](#what-is-generative-ai-computer-code) Generative AI computer code is code that an artificial intelligence model, such as a neural network, produces from a natural language prompt. These models pick up the structure and the small conventions of code from large training sets, so what they produce looks close to what a human programmer would write. The output runs from simple functions to complex algorithms, depending on the prompt and on what the model was trained on. ## [How do I create a Generative AI for code?](#how-do-i-create-a-generative-ai-for-code) Now the practical steps for building your Code Generative AI model. We’ll go through the process step by step so you can build your own AI code-writing assistant. ### [Step 1: Environment setup](#step-1-environment-setup) To start, you’ll need a Python environment with the libraries and dependencies it needs. The code below is a basic setup for your AI model, with the imports and the configuration settings: dumpster\_copilot\_generative.py ``` 1import logging2import torch3import peft4import transformers5from transformers import AutoTokenizer, AutoModelForCausalLM6from huggingface_hub.hf_api import HfFolder7 8# Configuration class for the Dumpster Copilot Generative model9class Configuration:10 ACCESS_TOKEN = 'ENTER YOUR HUGGINGFACE ACCESS TOKEN HERE'11 LOAD_IN_8BIT = False12 BASE_MODEL = 'meta-llama/Llama-2-7b-chat-hf'13 LORA_WEIGHTS = 'qblocks/llama2-7b-tiny-codes-code-generation'14 PROMPT = 'Write a Python function to divide 2 numbers and check for division by zero.'15 16# Exception classes for errors loading the model and generating text17class ModelLoadingError(Exception):18 pass19 20class DumpsterCopilotGenerativeError(Exception):21 pass22 23# Model loader class24class ModelLoader:25 @staticmethod26 def load_model() -> tuple:27 try:28 tokenizer = AutoTokenizer.from_pretrained(Configuration.LORA_WEIGHTS)29 model = AutoModelForCausalLM.from_pretrained(30 Configuration.BASE_MODEL,31 device_map='auto',32 torch_dtype=torch.float16,33 load_in_8bit=Configuration.LOAD_IN_8BIT34 )35 model = peft.PeftModel.from_pretrained(model, Configuration.LORA_WEIGHTS)36 return tokenizer, model37 except Exception as e:38 raise ModelLoadingError(f'Error loading tokenizer and model: {str(e)}') ``` In this code snippet, we’ve imported the libraries we need, `transformers` and `torch`, and added a `Configuration` class to hold the settings. The `ModelLoader` class is the one that loads the AI model. ### [Step 2: Loading your AI model](#step-2-loading-your-ai-model) Now that your environment is set up, it’s time to load your Code Generative AI model. In the code snippet, we’ve defined a `ModelLoader` class with a `load_model` method that handles the loading. This method returns a tokenizer and a model instance. Remember to replace `'ENTER YOUR HUGGINGFACE ACCESS TOKEN HERE'` with your actual Hugging Face access token. ### [Step 3: Generating code with your AI](#step-3-generating-code-with-your-ai) With your model loaded, you can generate code snippets with your AI assistant. The `DumpsterCopilotGenerative` class below runs the generation from a prompt you pass in: dumpster\_copilot\_generative.py ``` 1class DumpsterCopilotGenerative:2 def __init__(self, tokenizer, model):3 self.tokenizer = tokenizer4 self.model = model5 6 def dumpster_copilot_generative(self, prompt: str) -> str:7 try:8 logging.info(f'Generating text for prompt: {prompt}')9 10 generator = transformers.pipeline(11 'text-generation',12 model=self.model,13 tokenizer=self.tokenizer14 )15 16 generation_config = transformers.GenerationConfig(17 temperature=0.4,18 top_p=0.99,19 top_k=40,20 num_beams=2,21 max_new_tokens=400,22 repetition_penalty=1.323 )24 25 t = generator(prompt, generation_config=generation_config)26 27 generated_text = t[0]['generated_text']28 29 logging.info(f'Generated text: {generated_text}')30 return generated_text31 except Exception as e:32 raise DumpsterCopilotGenerativeError(f'Error generating text: {str(e)}') ``` The class has one method, `dumpster_copilot_generative`, which takes a prompt as input and returns the generated code. What you get back depends on the prompt, so make the prompt explicit and specific. ### [Step 4: Running your Code Generative AI](#step-4-running-your-code-generative-ai) Now that all the pieces are in place, you can run your Code Generative AI model and generate code. Here’s how: dumpster\_copilot\_generative.py ``` 1if __name__ == '__main__':2 try:3 if Configuration.ACCESS_TOKEN:4 HfFolder.save_token(Configuration.ACCESS_TOKEN)5 6 logging.info('Initiating the text generation process.')7 8 tokenizer, model = ModelLoader.load_model()9 generator = DumpsterCopilotGenerative(tokenizer, model)10 generated_text = generator.dumpster_copilot_generative(Configuration.PROMPT)11 12 logging.info('Generated text:')13 logging.info(generated_text)14 15 logging.info('Successful completion of the text generation process.')16 except (ModelLoadingError, DumpsterCopilotGenerativeError) as e:17 logging.error(f'An error occurred: {str(e)}') ``` This block initializes your AI model, generates code from the prompt (in this case, “Write a Python function to divide 2 numbers and check for division by zero.”), and logs the resulting code. ## [Frequently asked questions](#frequently-asked-questions) Now that you have the basics of building a Code Generative AI model, here are some questions that come up often. ### [Q1: How does Generative AI understand programming languages?](#q1-how-does-generative-ai-understand-programming-languages) Generative AI models learn programming languages by training on large amounts of code written in them. They pick up the syntax, the semantics, and the recurring patterns from many datasets, so the code they produce follows the conventions of the language you asked for. ### [Q2: Can Generative AI replace human programmers?](#q2-can-generative-ai-replace-human-programmers) Generative AI can automate parts of coding, such as writing boilerplate or completing code as you type. It does not replace human programmers. You still need people to design complex algorithms, to debug, and to make the decisions that matter in software development. ### [Q3: Are there ethical concerns related to AI-generated code?](#q3-are-there-ethical-concerns-related-to-ai-generated-code) Yes, there are. Two of them: bias in the training data, and people using generated code for harmful purposes. Use AI-generated code with care, and check that what it hands you meets your ethical standards. ### [Q4: What are some practical applications of Code Generative AI?](#q4-what-are-some-practical-applications-of-code-generative-ai) Code Generative AI is useful for code autocompletion, code refactoring, generating documentation, and helping with code reviews. It can make developers faster and it can help with code quality. ### [Q5: How can I fine-tune my Code Generative AI model?](#q5-how-can-i-fine-tune-my-code-generative-ai-model) Fine-tuning a Code Generative AI model means training it further on a specific dataset or domain so it gets better at that narrower job. You fine-tune an existing model with transfer learning techniques and domain-specific data. ## [Conclusion](#conclusion) In software engineering, AI, particularly Code Generative AI, could change how developers write code. By following the steps in this article, you can build your own Code Generative AI model to assist with your coding work. That said, AI is a tool, and human expertise and ethical considerations should still guide software development. Building your own Code Generative AI shows you exactly where it helps in your work and where it doesn’t. ## [References](#references) 1. Hugging Face Transformers Documentation, [https://huggingface.co/docs/transformers/index](https://huggingface.co/docs/transformers/index) 2. PEFT - Parameter-Efficient Fine-Tuning of Billion-Scale Models on Low-Resource Hardware, [https://github.com/huggingface/peft](https://github.com/huggingface/peft) 3. Llama 2 - Meta AI, [https://ai.meta.com/llama/](https://ai.meta.com/llama/) 4. “Attention Is All You Need” (Transformer paper) - Vaswani, A., et al. (2017.), [https://arxiv.org/abs/1706.03762](https://arxiv.org/abs/1706.03762) 5. “Evaluating Large Language Models Trained on Code” - Chen, M., et al. (OpenAI Codex.), [https://arxiv.org/abs/2107.03374](https://arxiv.org/abs/2107.03374) 6. Hugging Face Model Hub, [https://huggingface.co/models](https://huggingface.co/models) (Specifically search for code generation models like `meta-llama/Llama-2-7b-chat-hf` and `qblocks/llama2-7b-tiny-codes-code-generation`) 7. PyTorch Official Website, [https://pytorch.org/](https://pytorch.org/) Was this useful? ## Tags [#Generative AI](/blog/tags/generative-ai)[#Code Generation](/blog/tags/code-generation)[#AI in Software Development](/blog/tags/ai-in-software-development)[#Machine Learning Models](/blog/tags/machine-learning-models)[#Deep Learning for Code](/blog/tags/deep-learning-for-code)[#Hugging Face Transformers](/blog/tags/hugging-face-transformers)[#Python AI](/blog/tags/python-ai)[#Automation in Coding](/blog/tags/automation-in-coding)[#AI Ethics](/blog/tags/ai-ethics) ## Share [Facebook](https://facebook.com/sharer/sharer.php?u=https%3A%2F%2Fmkabumattar.com%2Fblog%2Fpost%2Fbuilding-a-code-generative-ai-model "Share on Facebook")[Twitter](https://twitter.com/intent/tweet/?text=Building%20a%20Code%20Generative%20AI%20Model&url=https%3A%2F%2Fmkabumattar.com%2Fblog%2Fpost%2Fbuilding-a-code-generative-ai-model "Share on Twitter")[LinkedIn](https://www.linkedin.com/shareArticle?mini=true&url=https%3A%2F%2Fmkabumattar.com%2Fblog%2Fpost%2Fbuilding-a-code-generative-ai-model&title=Building%20a%20Code%20Generative%20AI%20Model&summary=How%20to%20build%20a%20Code%20Generative%20AI%20model%20as%20a%20software%20engineer%3A%20how%20AI%20writes%20code%2C%20step-by-step%20instructions%20to%20build%20your%20own%2C%20and%20answers%20to%20common%20questions%20about%20AI-generated%20code.&source=https://mkabumattar.com "Share on LinkedIn")[WhatsApp](https://wa.me/?text=Building%20a%20Code%20Generative%20AI%20Model%20https%3A%2F%2Fmkabumattar.com%2Fblog%2Fpost%2Fbuilding-a-code-generative-ai-model "Share on WhatsApp")[Telegram](https://t.me/share/url?url=https%3A%2F%2Fmkabumattar.com%2Fblog%2Fpost%2Fbuilding-a-code-generative-ai-model&text=Building%20a%20Code%20Generative%20AI%20Model "Share on Telegram")[Reddit](https://www.reddit.com/submit?url=https%3A%2F%2Fmkabumattar.com%2Fblog%2Fpost%2Fbuilding-a-code-generative-ai-model&title=Building%20a%20Code%20Generative%20AI%20Model "Share on Reddit")[Hacker News](http://news.ycombinator.com/submitlink?u=https%3A%2F%2Fmkabumattar.com%2Fblog%2Fpost%2Fbuilding-a-code-generative-ai-model&t=Building%20a%20Code%20Generative%20AI%20Model "Share on Hacker News")[Pinterest](https://pinterest.com/pin/create/button/?url=https%3A%2F%2Fmkabumattar.com%2Fblog%2Fpost%2Fbuilding-a-code-generative-ai-model&media=&description=How%20to%20build%20a%20Code%20Generative%20AI%20model%20as%20a%20software%20engineer%3A%20how%20AI%20writes%20code%2C%20step-by-step%20instructions%20to%20build%20your%20own%2C%20and%20answers%20to%20common%20questions%20about%20AI-generated%20code. "Share on Pinterest")[Email]() ## Comments ## You might also enjoy More posts on similar topics [![Understanding Generative AI in Depth](/_astro/hero.DsJ5VDjO_ZYY4n1.webp)](/blog/post/understanding-generative-ai-in-depth) ## [Understanding Generative AI in Depth](/blog/post/understanding-generative-ai-in-depth) - [Mohammad Abu Mattar](/authors/mohammad-abu-mattar) - [Artificial Intelligence](/blog/categories/artificial-intelligence) - [Generative AI](/blog/categories/generative-ai) - [Machine Learning](/blog/categories/machine-learning) - [Deep Learning](/blog/categories/deep-learning) - [Software Engineering](/blog/categories/software-engineering) Introduction Artificial intelligence keeps changing fast, and senior software engineers need to keep up with emerging technologies. One technology that has gotten a lot of attention in recent year [#Generative AI](/blog/tags/generative-ai)[#GenAI](/blog/tags/genai)[#Machine Learning](/blog/tags/machine-learning)+9 tags [read more](/blog/post/understanding-generative-ai-in-depth) [![AI is Not Real: A Software Engineering Perspective](/_astro/hero.zLRxEs_v_7tKSA.webp)](/blog/post/ai-is-not-real) ## [AI is Not Real: A Software Engineering Perspective](/blog/post/ai-is-not-real) - [Mohammad Abu Mattar](/authors/mohammad-abu-mattar) - [Artificial Intelligence](/blog/categories/artificial-intelligence) - [Software Engineering](/blog/categories/software-engineering) - [Machine Learning](/blog/categories/machine-learning) - [Technology Ethics](/blog/categories/technology-ethics) We have all seen the wave of hype around artificial intelligence. It is everywhere, from tech conferences to science fiction scripts. As software engineers, though, we need to look past the marketing [#AI Limitations](/blog/tags/ai-limitations)[#Large Language Models](/blog/tags/large-language-models)[#Machine Learning](/blog/tags/machine-learning)+5 tags [read more](/blog/post/ai-is-not-real) [![GraphRAG Explained: Building Knowledge-Grounded LLM Systems](/_astro/hero.BdGCX8ya_1gL61X.webp)](/blog/post/graphrag-explained-building-knowledge-grounded-llm-systems) ## [GraphRAG Explained: Building Knowledge-Grounded LLM Systems](/blog/post/graphrag-explained-building-knowledge-grounded-llm-systems) - [Mohammad Abu Mattar](/authors/mohammad-abu-mattar) - [Artificial Intelligence](/blog/categories/artificial-intelligence) - [Machine Learning](/blog/categories/machine-learning) - [Large Language Models](/blog/categories/large-language-models) - [Knowledge Graphs](/blog/categories/knowledge-graphs) - [RAG Systems](/blog/categories/rag-systems) The world of artificial intelligence is moving fast. We've gone from being amazed that Large Language Models can write a poem to wanting them to be deeply grounded in factual truth. While these models [#GraphRAG](/blog/tags/graphrag)[#RAG](/blog/tags/rag)[#LLM](/blog/tags/llm)+7 tags [read more](/blog/post/graphrag-explained-building-knowledge-grounded-llm-systems) [![Microsoft's Prompt Orchestration Markup Language (POML): Structuring the Future of AI Interaction](/_astro/hero.ByJZRKLq_ZU3pnd.webp)](/blog/post/microsoft-poml-orchestrating-ai-prompts-for-llms) ## [Microsoft's Prompt Orchestration Markup Language (POML): Structuring the Future of AI Interaction](/blog/post/microsoft-poml-orchestrating-ai-prompts-for-llms) - [Mohammad Abu Mattar](/authors/mohammad-abu-mattar) - [AI & Machine Learning](/blog/categories/ai--machine-learning) - [Developer Tools](/blog/categories/developer-tools) - [Software Engineering](/blog/categories/software-engineering) Introduction: what is Microsoft's POML and why does it matter for AI? Large Language Models, or LLMs, are changing fast, and they're becoming super important for all sorts of complex applications. [#Microsoft](/blog/tags/microsoft)[#POML](/blog/tags/poml)[#Prompt Engineering](/blog/tags/prompt-engineering)+9 tags [read more](/blog/post/microsoft-poml-orchestrating-ai-prompts-for-llms) [![Understanding Infrastructure as Code (IaC)](/_astro/hero.D6yb7pOQ_Z2ptmwB.webp)](/blog/post/understanding-infrastructure-as-code-iac) ## [Understanding Infrastructure as Code (IaC)](/blog/post/understanding-infrastructure-as-code-iac) - [Mohammad Abu Mattar](/authors/mohammad-abu-mattar) - [Infrastructure as Code](/blog/categories/infrastructure-as-code) - [DevOps](/blog/categories/devops) - [Cloud Computing](/blog/categories/cloud-computing) - [Automation](/blog/categories/automation) - [Software Engineering](/blog/categories/software-engineering) Introduction Infrastructure as Code (IaC) manages infrastructure through code instead of manual configuration. Instead of clicking through consoles to set up servers, networks, and storage, you de [#IaC](/blog/tags/iac)[#AWS CloudFormation](/blog/tags/aws-cloudformation)[#Terraform](/blog/tags/terraform)+7 tags [read more](/blog/post/understanding-infrastructure-as-code-iac) [![REST API vs RESTful API: Architecture and Constraints Explained](/_astro/hero.D7ffsaFk_ZsjslT.webp)](/blog/post/rest-api-vs-restful-api-architecture-and-constraints-explained) ## [REST API vs RESTful API: Architecture and Constraints Explained](/blog/post/rest-api-vs-restful-api-architecture-and-constraints-explained) - [Mohammad Abu Mattar](/authors/mohammad-abu-mattar) - [API Development](/blog/categories/api-development) - [Web Architecture](/blog/categories/web-architecture) - [Software Engineering](/blog/categories/software-engineering) - [Backend Development](/blog/categories/backend-development) Introduction REST API and RESTful API get used interchangeably, but they aren't quite the same thing. This post covers the difference, REST's constraints, and what they mean for how you design an [#REST API](/blog/tags/rest-api)[#RESTful API](/blog/tags/restful-api)[#API Design Principles](/blog/tags/api-design-principles)+6 tags [read more](/blog/post/rest-api-vs-restful-api-architecture-and-constraints-explained) 6 related posts