As a Data Engineer at Creative CFO, I’ve witnessed firsthand how artificial intelligence is fundamentally reshaping SQL development, data engineering, and ultimately Business Intelligence. Over the past year, I’ve worked on both client-facing and internal products, using a combination of cutting-edge AI models and cloud-native tools to accelerate delivery, raise code quality, and build solutions that scale.
Here’s a detailed look at how we use AI for SQL in 2025, the strengths and weaknesses we’ve observed, and our recommendations for anyone looking to modernise their data engineering workflow.
The Benefits Of Using AI For Data Engineering
Implementing advanced AI models in our workflow has dramatically accelerated our development cycles. Tasks that once took hours of manual effort – like crafting SQL code, troubleshooting, or writing documentation — are now completed in a fraction of the time. This enables our team to focus on higher-level problem-solving and innovation rather than getting bogged down in repetitive scripting.
Another significant benefit has been the improvement in code quality and maintainability. By leveraging multiple models for cross-validation, we ensure that SQL code and data pipelines adhere to best practices and project standards. Tools like Cursor AI enable the proactive identification of performance bottlenecks, bugs, or security concerns before they reach production. Meanwhile, AI-powered meeting and video-conferencing transcription by Gemini keeps documentation accurate and up to date, making onboarding and compliance much easier.
Cost efficiency is also a significant advantage. With AI-driven performance tuning and automated code optimisation — especially in cloud environments like BigQuery — we minimise wasted resources and keep operational expenses in check. Docker and FastAPI further support this by allowing us to scale solutions seamlessly and control deployment costs as our projects grow.
However, these benefits come with important caveats. Human oversight remains essential. AI models, while powerful, are not perfect, they can make mistakes, misinterpret requirements, or produce inefficient solutions, especially in nuanced or edge-case scenarios. Every AI-generated output must be reviewed and, if necessary, refined by an experienced engineer before deployment. Over-reliance on AI can also risk eroding critical problem-solving skills, so it’s essential to maintain a strong balance between automation and manual expertise.
In summary, AI-powered SQL development and data engineering offer substantial improvements in speed, quality, and scalability. The key is thoughtful implementation — choosing the right models, providing accurate context, and maintaining a layer of human oversight at all times. For teams willing to invest in this balance, the rewards are considerable: faster delivery, smarter solutions, and future-proof data platforms for both clients and internal stakeholders.
Testing Different AI models
Our process begins by supplying AI models with up-to-date project documentation and context, ensuring that every code suggestion, business logic, or output is relevant and of high quality. AI-generated code is constantly reviewed and refined before deployment/implementation. Tools like Docker and FastAPI guarantee our solutions are portable and scalable, while BigQuery and Cursor AI help us maintain performance and quality as projects evolve and grow.
At Creative CFO, we rely on Team GPT, which brings together several of the most advanced AI models for data engineering:
- OpenAI: GPT-4, GPT-4o, GPT-3.5, GPT-3.5 Turbo, o1, o1-mini
- Anthropic: Claude 3 Opus (and Opus 4), Claude 3 Sonnet, Claude 3 Haiku
- Google: Gemini 1.5 Pro, Gemini 1.5 Flash
With Team GPT, we can upload project documentation, data dictionaries, and notes, allowing the models to generate code and recommendations that truly reflect our business requirements and technical context.
When it comes to testing different models for various tasks, we’ve done so across several client and internal projects to determine the best models and their applications.
Using AI models for Client Project Success
Our engagement with a prominent financial firm required building a flexible and reliable SQL database, along with numerous ETL pipelines capable of keeping pace with evolving requirements and stringent quality standards.
By feeding all relevant documentation and data definitions into Team GPT, we enabled the AI to generate SQL tables, stored procedures, and jobs that met our standards from the outset. This approach allowed us to quickly adapt to changing business rules, reduce manual coding, and maintain consistency as the project evolved.
Key tools used:
Team GPT (AI Model Platform)
Pros:
- Rapid, context-aware SQL and code generation.
- Consistency in naming conventions and logic.
- Automates repetitive coding and documentation tasks.
- Adapts quickly to business and technical feedback.
- Enhances error handling and code reusability.
Cons:
- The output can be incorrect or inefficient if the context is incomplete.
- May miss nuanced business requirements.
- Sensitive data must be carefully managed.
- Can suggest over-engineered solutions.
Google Gemini (Transcription & Summarization)
Pros:
- Fast, accurate transcription of meetings.
- Reduces the risk of missing requirements or decisions.
- Streamlines documentation and project communication.
- Supports onboarding and knowledge transfer.
Cons:
- Can misinterpret jargon or strong accents.
- Subtle meeting context may be missed unless reviewed.
SQL Server Management Studio (SSMS)
Pros:
- Enterprise-grade tools for SQL Server development.
- Comprehensive management and deployment features.
- Strong security and compliance support.
- Excellent integration with existing MS infrastructure.
Cons:
- Higher licensing and infrastructure costs.
- Slower prototyping compared to cloud-native tools.
- More rigid workflows than serverless options.
Using AI models for Internal Product Development
When architecting our internal data platform, our goals were scalability, rapid development, and seamless integration with multiple data sources. Here, containerization, API-first design, and serverless data warehousing were critical, with AI tools supporting code generation, analysis, and documentation at every step.
Key tools used:
Docker (Containerization & Orchestration)
Pros:
- Guarantees consistent deployments across environments.
- Lightweight, efficient use of resources.
- Simplifies scaling and rollback for microservices.
- Supports modular architecture.
Cons:
- Orchestration and setup can be complex.
- Requires DevOps expertise.
- Debugging inside containers can be challenging.
FastAPI (API Development)
Pros:
- Extremely fast to develop and run.
- Automatic, interactive API documentation (Swagger, ReDoc).
- Strong data validation and error handling.
- Native support for async operations.
Cons:
- Smaller ecosystem than some established frameworks.
- Async code can increase debugging complexity.
- Some advanced features are still maturing.
BigQuery (Cloud Data Warehouse)
Pros:
- Fully managed, scalable, and serverless.
- Pay-per-use model controls infrastructure costs.
- Supports real-time analytics and ingestion.
- Integrates seamlessly with the Google Cloud ecosystem.
Cons:
- Costs can escalate if queries aren’t optimised.
- Vendor lock-in risk.
- Learning curve for new users.
Cursor AI (Codebase Analysis & Suggestions)
Pros:
- Proactively analyses the whole codebase for quality and security.
- Offers actionable suggestions for improvement.
- Helps maintain standards as the codebase grows.
Cons:
- Not always context-aware — manual review still needed.
- May miss project-specific nuances.
- Sometimes recommends generic fixes.
Our AI Model Recommendations
After extensive testing in real-world projects, Creative CFO recommends the following AI models for SQL and data engineering in 2025, each selected for its particular strengths:
Claude 3 Sonnet (Anthropic)
Best For: Context-rich SQL tasks, complex queries, and projects with evolving business logic.
Why: Excels at digesting detailed documentation and generating code that aligns with both technical and business requirements. Its reasoning abilities are especially valuable when requirements change, or the context is complex.
OpenAI GPT-4o
Best For: Rapid prototyping, multi-language code generation, and debugging.
Why: Highly versatile and fast, GPT-4o is ideal for generating SQL, Python, or documentation in iterative cycles. It’s powerful for troubleshooting and quickly building out new features.
Gemini 1.5 Pro (Google)
Best For: Transcribing meetings, summarising requirements, and maintaining project documentation.
Why: Automates the capture of discussions and decisions, ensuring nothing gets lost and making onboarding and compliance much more straightforward.
Cursor AI (Codebase Analysis & Suggestions)
Best For: Automated code reviews, holistic codebase analysis, and ongoing code quality improvements.
Why: Cursor AI provides proactive, context-aware suggestions for improving code quality, identifying security risks, and maintaining standards as projects scale. It’s particularly effective for teams that want to keep their codebase robust and future-proof through continuous improvement.
Each of these models brings unique strengths to the table, and by combining them, our team delivers solutions that are both robust and agile.
Looking Forward
The integration of AI tools has completely transformed our workflow at Creative CFO. Solutions are built faster, scale more easily, and are better documented – benefits that extend far beyond SMEs to any business or industry seeking to modernise its data engineering or even other areas of business. As AI models continue to improve, we expect their impact on quality, innovation, and efficiency to grow even further.
