title: 📚 The Smart Way to Talk to Your Database: Why Hybrid API + NL2SQL Wins? published: true date: 2025-05-07 21:45:19 UTC tags: ai,database,sql,api

canonical_url: https://dbconvert.com/blog/hybrid-nl2sql-vs-full-ai/

Hybrid is not a fallback — it's the real strategy.

Why Pure AI Isn't Enough — And How Combining Bulletproof APIs with Smart NL2SQL Creates the Future of Database Interaction

1. Introduction

Databases weren't designed to "listen" — meaning understand flexible human intentions. They were designed to "obey" — meaning strictly execute SQL commands. Now it's time to teach them both.

For decades, database systems have been built on strict, predictable APIs: list your /tables, fetch /meta, run SELECT queries — and everything just works.

But today, with AI evolving rapidly, a powerful new dream is emerging:

"Can users finally talk to databases in natural language — no SQL textbooks, no syntax memorization, just questions?"

Yet reality bites: AI alone can't replace strong backend architecture.

The real solution? A Hybrid Approach — traditional bulletproof APIs + an AI-powered NL2SQL layer (Natural Language to SQL) that acts as an optional bonus.

Let's break it down — pragmatically, not dreamily.

2. Why Pure AI Won't Cut It (Yet)

Reality check:

• You don't want critical operations depending only on AI "best guesses."
• You DO want natural language as a bonus layer — not just for non-technical users, but for anyone who values saving time and riding the new wave of 'vibe coding' that's spreading fast.

Thus: Hybrid wins. It's smarter, faster, and cooler — because it actually works. And as a result, it's way sexier than blind "AI magic."

Even the most advanced AI database tools today rely on strong traditional APIs underneath. There are no magic shortcuts — robust backend foundations are non-negotiable.

3. Hybrid Architecture Blueprint

Frontend (UI)
   ↓
Backend (Traditional APIs)
   ↓
• /meta (List tables, views)
• /tables (Detailed table info)
• /views (View info)
• /execute (Safe SELECT/SHOW only)
   ↓
NL2SQL Layer (Optional, AI-assisted)
   ↓
Smart prompt ➔ OpenAI (or local LLM)
   ↓
Return generated SQL
   ↓
Safe validate SQL
   ↓
Execute via /execute
   ↓
Results to User

hybrid architecture

4. Traditional Responsibilities

Your backend should ALWAYS handle:

• Schema serving: /meta, /tables, /views
• Safe query execution: /execute (read-only enforced)
• Connection pooling and auth
• Error handling and logging

These parts MUST NOT depend on any LLM.

Treat LLM as optional bonus.

5. AI/NL2SQL Responsibilities

AI should ONLY help:

• Translate user intent into SQL.
• Suggest queries based on partial language.
• Explore data more flexibly.

BUT:

• Validate generated SQL strictly.
• Never allow unsafe commands (e.g., DROP, DELETE).
• Rate-limit AI usage if needed to avoid abuse.

6. Prompt Engineering Example

You are an expert SQL assistant for a PostgreSQL database.
Here are the available tables:

- users (id, name, email)
- orders (id, user_id, total_amount, created_at)

Instructions:
- Generate a single-line SQL query (PostgreSQL syntax).
- Use only the provided tables and columns.
- Format output like this:

```sql
SELECT * FROM users;

User Question: List all users who placed an order over $500.

Example SQL generated:

```sql
SELECT users.*
FROM users
JOIN orders ON users.id = orders.user_id
WHERE orders.total_amount > 500;

Result: Clean, focused, safe query generation.

7. Conclusion: Brains Over Buzzwords

✅ Backend: solid, predictable, safe.

✅ AI layer: flexible, optional, user-friendly.

Don't throw away proven API design. Don't fear adding smart, lightweight AI layers.

Be pragmatic. Combine them.

That's how real production systems win.

7.5 Why Hybrid Saves You from Catastrophes

Some dreamers imagine this:

"I'll just send the entire multi-million-row table to the AI and let it figure things out."

Reality check:

• 🚫 LLMs can't handle massive raw data ingestion (token limits, timeouts, costs skyrocket).
• 🚫 It's dumb to flood AI with 100+MB payloads.
• 🚫 You destroy speed, efficiency, and security in the process.

**Hybrid solves it differently:

• ✅ Use traditional APIs (/meta, /sample, /aggregate, /data) to pre-filter, slice, and intelligently fetch only needed records.
• ✅ Only send small, smart prompts to AI — let it generate smart queries, not drown in raw data.

Even when building AI-driven systems, never let your LLM blindly query raw data. Always use traditional API endpoints (/meta, /sample, /aggregate, /data) to prepare clean, small context before AI gets involved.

Small context = Smart answers. Big chaos = Dumb crashes.

In short: AI thinks better when you feed it knowledge — not raw chaos.

🧪 DBConvert Streams: Real Tools for Real Builders

As of version 1.3, DBConvert Streams already provides everything you need to power the hybrid approach:

• ✅ View full database structure
• ✅ Fetch table data cleanly
• ✅ Inspect DDL for tables and views via API

And yes — we're not stopping there. NL2SQL is coming very soon in the next release.

Stay tuned.

Build smarter, connect deeper — and leave the AI noise merchants behind.

Final thought: In a world chasing AI hype, it's those who blend power with precision who build systems that truly last.

title: Stop Using pgloader: This No-Code Tool Migrates to Neon Faster published: true date: 2025-04-15 22:21:45 UTC tags: Neon,postgresql,postgres,mysql

canonical_url: https://dbconvert.com/blog/stop-using-pgloader-this-no-code-tool-migrates-to-neon-faster/

#

As developers and teams look for scalable, flexible database infrastructure, Neon emerges as a modern serverless PostgreSQL platform. It offers features like autoscaling, branching, and separation of storage and compute—making it ideal for cloud-native apps.

While Neon's official documentation suggests using pgloader for migration from MySQL, that approach requires scripting and command-line tools.

For those looking for a no-code, streamlined experience, DBConvert Streams provides a powerful, real-time migration solution with a friendly web interface.

Why Neon?

Neon provides a cloud-native PostgreSQL environment with:

• Autoscaling — scales compute resources automatically based on demand.
• Branching — spin up isolated database branches instantly for dev/test.
• Separation of compute and storage — optimized for elasticity and performance.
• Pay-as-you-go pricing — ideal for startups and scaling workloads.

🔄 What Is DBConvert Streams?

DBConvert Streams is a no-code, real-time database migration and replication tool. It supports migrating from MySQL to PostgreSQL - perfect for transferring data to Neon from any MySQL source, whether it's hosted locally, self-hosted on your own servers, running on cloud platforms (AWS, GCP, Azure), or on managed database services like Amazon RDS or Google Cloud SQL.

Key Features

• ✅ Web UI — no CLI or code needed.
• ✅ Real-time CDC (Change Data Capture) sync.
• ✅ Automated schema mapping and transformation.
• ✅ Deployable locally or to the cloud (DigitalOcean, AWS, etc.).

🛠️ Prerequisites

Before starting the migration, make sure you have:

1. Local MySQL database :
   • Host: localhost
   • Port: 3306 (default)
   • Credentials (user, password)
   • Database name
2. Neon PostgreSQL instance :
   • Sign up at neon.tech
   • Create a database project
   • Copy the PostgreSQL connection string from the Neon Console
3. Deployment of DBConvert Streams :
   • Go to: https://streams.dbconvert.com/deploy
   • Choose your preferred deployment (Docker, local binary, or cloud)

🔧 Step-by-Step Migration Process

1. Launch DBConvert Streams

After deployment:

• Open the web interface at:
  http://localhost (for local)
  or
  http://<your_server_ip> (for cloud-hosted)

2. Set Up Source and Target

• Source :
  Choose MySQL , and enter your local database details:
  host=localhost, port=3306, username=root, password=yourpassword, database=mydb
• Target :
  Choose PostgreSQL , and paste your Neon connection string.

3. Create a Migration Stream

• Select the tables you want to migrate.
• Choose "One-time migration" for transferring data to Neon.
• Start the stream.

stream to migrate data from local MySQL to Neon

4. Monitor Progress

• Use the dashboard to track row-level sync.
• Logs and statistics are available in real time.

5. Verify in Neon

• Connect to your Neon database using psql, or Neon's Viewer.
• Verify schema, data, and constraints.

📊 Real-World Example: Migrating the Sakila Database

To demonstrate the power and simplicity of DBConvert Streams, let's look at migrating the standard Sakila sample database (a widely-used sample database that models a DVD rental store).

The pgloader Approach (Command Line)

Using pgloader requires creating a configuration file like this:

LOAD DATABASE
    FROM mysql://root:password@mysql-source/sakila?useSSL=false
    INTO pgsql://neondb_owner:endpoint=endpoint;PasSwOrD@addr.eu-central-1.aws.neon.tech/neondb?sslmode=allow
WITH include drop, 
     create tables, 
     create indexes, 
     reset sequences,
     workers = 4, 
     concurrency = 1,
     multiple readers per thread, 
     rows per range = 50000,
     drop indexes
SET MySQL PARAMETERS
    net_read_timeout = '120',
    net_write_timeout = '120'
CAST
    type datetime to timestamptz drop default using zero-dates-to-null,
    type timestamp to timestamptz drop default using zero-dates-to-null,
    type date to date drop default using zero-dates-to-null,
    type tinyint to smallint drop typemod,
    /* and many more type mappings... */

pgloader load.config

This requires understanding data type mappings, SQL specifics, and executing commands via CLI.

The DBConvert Streams Approach (No-Code)

With DBConvert Streams, the process is dramatically simplified:

1. Select MySQL source and enter connection details for Sakila database
2. Select PostgreSQL target and enter Neon connection string
3. Click to select all tables
4. Start the migration

💡

The results?
pgloader : ~13 seconds with manual configuration
DBConvert Streams : ~1 second with zero configuration

While both tools successfully migrated the data, DBConvert Streams did it:

• Without requiring coding knowledge
• Without manual type mapping configuration
• In a fraction of the time
• Through an intuitive web interface

This real-world test demonstrates how DBConvert Streams removes complexity while delivering superior performance for database migrations.

🧩 Why Not Use pgloader?

While Neon's official docs recommend pgloader, it's:

• CLI-based and less intuitive for non-technical users
• Requires manual configuration of data type mappings
• Lacks real-time sync support
• Not ideal for visual monitoring or production replication
• Generally slower for most migration scenarios

DBConvert Streams provides a visual, no-code, production-friendly alternative with CDC and web-based control.

Need to migrate from AWS RDS, Google Cloud SQL, or DigitalOcean? Stay tuned — support’s just as easy.
🌐 Visit https://streams.dbconvert.com/deploy to get started.

Say goodbye to scripts and hello to seamless, serverless PostgreSQL with Neon.

title: "Escape Google Cloud SQL: Migrate to DigitalOcean with MySQL or PostgreSQL (No-Code Guide)" published: true date: 2025-04-13 20:18:08 UTC tags:

canonical_url: https://dbconvert.com/blog/migrate-mysql-postgresql-to-digitalocean/

#

DigitalOcean is a favorite among developers, indie hackers, and growing startups for its clean developer experience, predictable pricing, and powerful managed services. It's especially appealing to those who want to escape the complexity and overhead of platforms like Google Cloud.

If you're currently hosting your database on Google Cloud SQL and find yourself overwhelmed by its enterprise-focused tooling or rising costs, moving your data to DigitalOcean is a smart move. With managed PostgreSQL and MySQL offerings, DigitalOcean makes it easy to offload operational tasks like backups, updates, and high availability—so you can focus on building your product.

In this guide, I'll show you how to transfer your database from Google Cloud SQL to DigitalOcean using DBConvert Streams — a powerful no-code tool designed to simplify cross-database migrations across cloud environments. For this demo, we will extract data from a MySQL source database and transfer it to a PostgreSQL target database.

Prerequisites

Before you begin, make sure you have the following ready:

• ✅ A DigitalOcean account (needed to create both a Droplet for DBConvert Streams and a target Postgres managed database)
• ✅ Credentials for your Google Cloud SQL instance (host, port, username, password, database name)

Step 1: Set Up Your DigitalOcean Database

DigitalOcean offers fully managed MySQL and PostgreSQL databases that are perfect for production workloads.

1. Log in to your DigitalOcean dashboard
2. Click on Databases in the left menu
3. Click Create Database Cluster
4. Select your database engine (in our PostgreSQL)
5. Choose your preferred plan (start small - you can scale up later)
6. Select the datacenter region closest to your users
7. Give your database a name (e.g., "sakila-db")
8. Click Create Database Cluster

DigitalOcean will take a few minutes to provision your database. Once ready, you'll see connection details including hostname, port, username, password, and database name.

Step 2: Deploy DBConvert Streams 1-Click App

The simplest way to deploy DBConvert Streams is through the DigitalOcean Marketplace:

1. Visit https://marketplace.digitalocean.com/apps/dbconvert-streams
2. Click Create DBConvert Streams Droplet
3. Choose your plan (Basic Droplet with 2GB RAM is sufficient for most migrations)
4. Select the same region as your database for optimal performance
5. Add your SSH key or create a password
6. Click Create Droplet

Once your Droplet is provisioned (usually takes about a minute), you can access the DBConvert Streams web interface by navigating to http://<your-droplet-ip> in your browser.

Step 3: Configure Google Cloud SQL for External Access

Before creating a connection in DBConvert Streams, you must configure your Google Cloud SQL instance to accept external connections:

1. In the Google Cloud Console, navigate to your Cloud SQL instance
2. Go to the Connections tab
3. Under Networking , select Add network in the Authorized networks section
4. Add your DigitalOcean Droplet's IP address with a name like "streams-migration"
5. Save your changes

For detailed instructions on configuring Google Cloud SQL for external access, refer to the Google Cloud SQL Connection Guide in the DBConvert Streams documentation.

###

Step 4: Obtain and Configure Your API Key

When you first open the DBConvert Streams web interface, you'll need to provide an API key:

1. Visit https://streams.dbconvert.com/account in another tab
2. Sign up or log in with your preferred authentication method
3. Copy your API key from the account dashboard
4. Return to your DBConvert Streams interface on your Droplet
5. Paste the API key and click Continue

get your API Key

This activates your free trial with 5GB of data transfer - more than enough for most small to medium database migrations.

DBConvert Streams offers a generous free trial that includes:

• 5GB of data transfer
• 14 days of unlimited access to all features
• No credit card required to start

This gives you ample opportunity to test the migration process before committing to a paid plan, making it risk-free to try for your DigitalOcean managed database migration project.

Step 5: Connect Your Source Database (Google Cloud SQL)

Now that your environment is set up, it's time to connect to your Google Cloud SQL database:

1. From the DBConvert Streams dashboard, click Create Connection
2. Select MySQL database type
3. Enter a descriptive name like "Google Cloud SQL - Sakila"
4. Enter the connection details from Google Cloud SQL:
   • Server : Your instance's public IP address
   • Port : 3306 for MySQL
   • User ID : Your database username
   • Password : Your database password
   • Database : Your database name
5. If using SSL (recommended), enable it and upload any certificates
6. Select Database.
7. Optionally click Test Connection to verify connectivity
8. Click Update

Step 6: Connect Your Target Database (DigitalOcean)

Next, set up the connection to your new DigitalOcean database:

1. Click Create Connection again
2. Select PostgreSQL database type
3. Enter a name like "DigitalOcean - Sakila"
4. Enter the DigitalOcean database connection details:
   • Server : The host shown in your DigitalOcean database connection details
   • Port : 25060 (DigitalOcean's default port for managed databases)
   • User ID : The default user (typically "doadmin")
   • Password : The password shown in your connection details
   • Database : Your database name
5. Enable SSL and select "Require" or "Verify-CA" mode
6. For PostgreSQL, you may need to specify the schema (usually "public")
7. Click Test Connection to verify
8. Click Update

add target db

two connections are available

Step 7: Configure Your Migration Stream

Now it's time to set up the actual migration process:

• Click Create Stream from the dashboard

Create New Stream

• Select your Google Cloud SQL connection as the source and click Next

• You'll see a list of tables from your database - select the ones you want to migrate (or Select All)
• Choose the data transfer mode:

– Convert/Migrate : Best for one-time migrations (what we need now)

– CDC/Stream : For continuous real-time replication (useful for zero-downtime migrations)

For our sample we choose Convert/Migrate mode

Select your DigitalOcean database as the target

Review your configuration and click Save

Step 8: Start and Monitor the Transfer

1. From the streams page, click Start

The dashboard will show real-time progress of your migration:

• Tables being processed
• Number of records transferred
• Transfer speed
• Estimated completion time

During the transfer, you can:

• View detailed logs to monitor the process
• Pause the transfer if needed (and resume later)
• Monitor system resources on your Droplet

For larger databases, you may want to set up Droplet monitoring in the DigitalOcean dashboard to ensure you have sufficient resources for the migration.

Step 9: Verify Your Data

Once the process completes, it's crucial to verify that your data has been transferred correctly:

1. Connect to your DigitalOcean database using the command line or a GUI tool like TablePlus, DBeaver, or DigitalOcean's built-in console:

PGPASSWORD=<password> psql -h <do-host> -p 25060 -U doadmin defaultdb

1. Run verification queries to check data integrity:

-- List all tables in the database
\dt

-- Get row counts for all tables
SELECT 
    schemaname as schema,
    relname as table_name,
    n_live_tup as row_count
FROM pg_stat_user_tables
ORDER BY n_live_tup DESC;

-- Check table sizes including indexes
SELECT
    table_schema,
    table_name,
    pg_size_pretty(pg_total_relation_size('"' || table_schema || '"."' || table_name || '"')) as total_size
FROM information_schema.tables
WHERE table_schema = 'public'
ORDER BY pg_total_relation_size('"' || table_schema || '"."' || table_name || '"') DESC;

1. Compare these results with the same queries run on your Google Cloud SQL instance to ensure all data has been transferred correctly.

1. Optionally run some more verification queries to make sure all data copied successfully. In your case, these queries will be according to your individual needs.

-- List all tables
\dt

-- Get row counts for key tables with sample data
SELECT 'customers' as table_name, COUNT(*) as row_count FROM customer
UNION ALL
SELECT 'films' as table_name, COUNT(*) as row_count FROM film
UNION ALL
SELECT 'rentals' as table_name, COUNT(*) as row_count FROM rental
UNION ALL
SELECT 'payments' as table_name, COUNT(*) as row_count FROM payment;

-- Verify data integrity with sample queries
-- Check total revenue
SELECT SUM(amount) as total_revenue FROM payment;

-- Check top 5 rented films
SELECT f.title, COUNT(*) as rental_count
FROM rental r
JOIN inventory i ON r.inventory_id = i.inventory_id
JOIN film f ON i.film_id = f.film_id
GROUP BY f.title
ORDER BY rental_count DESC
LIMIT 5;

-- Verify customer data
SELECT COUNT(*) as active_customers
FROM customer
WHERE active = 1;

Step 10: Update Your Application

Once you've verified your data, it's time to update your application to use the new DigitalOcean database:

1. Update your application's database connection settings:
   • Host: Your DigitalOcean database hostname
   • Port: 25060
   • Username: doadmin (or your custom user)
   • Password: Your database password
   • Database name: Your database name
   • SSL: Required (usually)
2. If you're using managed app platforms like DigitalOcean App Platform, update your environment variables with the new database connection details.

##

Why Choose DBConvert Streams for Your DigitalOcean Migration?

• User-friendly interface that doesn't require deep database expertise
• Automated schema conversion between different database types
• Cross-database conversion between PostgreSQL and MySQL
• Real-time monitoring of the migration process
• Flexible migration options including one-time transfers and continuous replication
• Secure credential management with encrypted connections

Conclusion

Migrating from Google Cloud SQL to DigitalOcean doesn't have to be complex or risky. With DBConvert Streams, you can perform the migration efficiently while maintaining data integrity. DigitalOcean's straightforward managed database offerings provide an excellent destination for your data, with predictable pricing and developer-friendly tools.

After completing this migration, you'll benefit from DigitalOcean's simplified database management, clear pricing structure, and performance optimized for developer workloads. Most importantly, you'll spend less time managing your database infrastructure and more time building your application.

Ready to start your migration? Visit https://marketplace.digitalocean.com/apps/dbconvert-streams to deploy the DBConvert Streams 1-Click App and experience the simplest way to move your database to DigitalOcean.

title: Announcing DBConvert Streams: First Public Release of Our Cloud-Native Database Migration Platform published: true date: 2025-02-24 11:48:47 UTC tags: database, replication, postgres, mysql

canonical_url: https://dbconvert.com/blog/dbconvert-streams-release/

Introduction

Database migration and synchronization have long been critical challenges for organizations managing data across different platforms. Whether moving from MySQL to PostgreSQL, setting up real-time data replication, or maintaining synchronized databases across different regions, these operations often involve complex processes, potential downtime, and significant technical overhead.

While DBConvert's existing solutions at dbconvert.com have served users well as Windows desktop applications, today's cloud-centric world demands a different approach. That's why we're excited to introduce DBConvert Streams, our newest addition to the DBConvert family of products. Built from the ground up as a cloud-native solution, DBConvert Streams runs natively on Linux environments - the dominant platform for modern hosting providers and cloud infrastructure. This first public release focuses on robust support for MySQL and PostgreSQL databases, delivering powerful capabilities for both one-time migrations and continuous real-time replication between these popular database platforms.

Core Capabilities

Universal Database Compatibility

DBConvert Streams handles data transfer between MySQL and PostgreSQL databases, regardless of where they're hosted. Our intelligent schema mapping automatically handles the complexity of different database types and cloud platforms.

Supported Databases:

• MySQL and compatible databases (MariaDB, SingleStore)
• PostgreSQL and compatible databases (CockroachDB)
• Cloud-managed databases:
  • Amazon RDS/Aurora (MySQL and PostgreSQL)
  • Google Cloud SQL
  • Azure Database
  • DigitalOcean Managed Databases

Whether you're migrating from MySQL to PostgreSQL, synchronizing between cloud providers, or replicating from cloud to on-premises, DBConvert Streams manages the entire process automatically.

Two Powerful Operating Modes

Real-time Change Data Capture (CDC)

DBConvert Streams leverages native database capabilities - MySQL's binary logs and PostgreSQL's Write-Ahead Logs (WAL) - to capture and replicate changes in real-time with:

• Zero-downtime operation
• Minimal source impact
• Immediate change propagation
• Complete capture of all data modifications

Fast Data Migration

For one-time transfers, DBConvert Streams employs intelligent chunking technology that:

• Optimizes large-scale transfers
• Provides real-time monitoring
• Ensures data consistency
• Automatically handles schema conversion

Built-in Monitoring and Security

Every aspect of data transfer is observable through comprehensive dashboards and metrics, while security is ensured through:

• SSL/TLS encryption for all database connections
• Secure credential management via HashiCorp Vault
• API key authentication
• Detailed audit logging

Technical Architecture

DBConvert Streams is built on a modern, distributed architecture designed for reliability and scalability. The platform consists of three main components:

1. API Server : Manages stream configurations and user interactions
2. Source Reader : Handles data extraction from source databases
3. Target Writer : Manages writing to destination databases

This architecture is supported by enterprise-grade infrastructure:

• NATS for reliable message streaming
• HashiCorp Vault for secure credential management
• Consul for service discovery and configuration

dbconvert streams architecture

Practical Use Cases

Real-time Analytics

• Replicate transaction data between OLTP and analytics platforms in any direction (MySQL to PostgreSQL or PostgreSQL to MySQL)
• Maintain live reporting systems with minimal latency, regardless of source or target database type
• Enable real-time business intelligence without impacting production databases

Cloud Migration

• Transfer data freely between any combination of cloud and on-premises databases
• Move data in any direction: cloud-to-cloud, cloud-to-on-premises, or on-premises-to-cloud
• Switch between cloud providers with automated schema conversion (e.g., AWS Aurora to Azure Database or vice versa)
• Create development environments with production data from any source to any target

System Modernization

• Transition from legacy MySQL systems to modern PostgreSQL platforms
• Maintain synchronized systems during gradual migrations
• Enable hybrid deployments during transition periods

Getting Started

DBConvert Streams offers flexible deployment options to suit different environments. Visit our deployment guide to choose between Docker containers or binary installation based on your needs.

After installation, the intuitive web interface guides you through:

1. Creating database connections
2. Configuring your first stream
3. Monitoring transfer progress
4. Managing ongoing operations

Visit our getting started guide for detailed instructions.

Modern Web Interface Benefits

The DBConvert Streams dashboard provides a modern web interface that offers:

• Real-time monitoring of data transfer progress and system metrics
• Visual configuration of database connections with instant validation
• Point-and-click stream setup without complex configuration files
• Comprehensive overview of all your streams in one place
• Access from any browser without installing desktop software
• Team-friendly interface for collaborative database operations

Future Development

While this initial release focuses on MySQL and PostgreSQL support, DBConvert Streams is built for expansion. Our roadmap includes:

• Support for additional database platforms
• Enhanced transformation capabilities
• Advanced monitoring features
• Expanded cloud integration options

Conclusion

DBConvert Streams represents a significant step forward in database migration and replication technology. By focusing initially on MySQL and PostgreSQL support, we've created a robust foundation that delivers immediate value while setting the stage for future expansion.

Start with a 10-day free trial that includes:

• 5GB data transfer limit during the trial period
• All features fully enabled
• Option to upgrade to full data transfer limits by adding payment details during trial
• No payment required during trial period

Ready to transform your database operations? Visit streams.dbconvert.com to start your free trial, or explore our comprehensive documentation at docs.dbconvert.com/streams to learn more.

title: ChatGPT: Your Guide to SQL Query Translation between Databases. published: true date: 2024-05-24 16:49:52 UTC tags: chatgpt,ai,sql,database

canonical_url: https://dbconvert.com/blog/using-chatgpt-for-sql-query-translation/

Introduction

Everyone knows that ChatGPT is perfect for translating between many human languages. But did you know that this powerful language model can also excel at converting SQL queries between various database dialects?

Whether you are transitioning from MySQL to PostgreSQL, SQL Server to Oracle, or any other combination, ChatGPT can assist in accurately translating your SQL queries. This capability extends beyond simple syntax changes, providing insights into how database systems handle data types, functions, and constraints. By leveraging ChatGPT for SQL translation, you can ensure a smoother and more efficient transition between database systems, maintaining data integrity and query performance.

Understanding the Challenge

Translating SQL queries between different database systems takes a lot of work. Each database system, be it MySQL, PostgreSQL, SQL Server, or Oracle, has its own distinct SQL dialect, encompassing specific syntax, functions, data types, and constraints. These variations can present substantial hurdles during migration.

Example1: Auto-Increment Columns

MySQL:

In MySQL, the AUTO_INCREMENT keyword defines an auto-incrementing primary key.

CREATE TABLE users (
    id INT AUTO_INCREMENT PRIMARY KEY,
    name VARCHAR(100)
);

PostgreSQL:

In PostgreSQL, you use SERIAL to auto-increment fields.

CREATE TABLE users (
    id SERIAL PRIMARY KEY,
    name VARCHAR(100)
);

SQL Server:

In SQL Server, the IDENTITY property defines an auto-incrementing primary key.

CREATE TABLE users (
id INT IDENTITY(1,1) PRIMARY KEY,
name NVARCHAR(100) NOT NULL
);

Oracle:

In Oracle, since version 12c, the IDENTITY Column method has been recommended.

CREATE TABLE users ( 
id NUMBER GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY, 
name VARCHAR2(100) 
);

In versions below 12c, Oracle uses a complex sequence and trigger mechanism.

Example 2: String Functions

MySQL:

The CONCAT_WS function in MySQL concatenates strings with a specified separator.

SELECT CONCAT_WS('-', first_name, last_name)
FROM users;

PostgreSQL:

In PostgreSQL, you can use the CONCAT function along with the separator.

SELECT CONCAT(first_name, '-', last_name)
FROM users;

Oracle:

You can achieve the same result in Oracle using the || operator for string concatenation.

SELECT first_name || '-' || last_name AS full_name
FROM users;

Additionally, there are variations in how each system handles transactions, error handling, and even indexing.

Moreover, some database systems include proprietary features that lack direct equivalents in other systems. This situation often makes straightforward translation impossible, requiring the development of alternative solutions or workarounds to achieve the same functionality.

Grasping these challenges is pivotal for a successful migration. It necessitates a profound understanding of the source and target database systems and the intricacies of their SQL dialects. This is where ChatGPT shines. With its extensive language model capabilities, it can help identify and tackle these differences, offering precise translations and guiding users through the intricacies of the transition process.

How ChatGPT Can Help

ChatGPT can be an invaluable tool for developers and database administrators tasked with migrating SQL queries and database structures between different systems. Here's how ChatGPT can assist in this process:

Accurate Query Translation.

ChatGPT excels at understanding the nuances of various SQL dialects. It can accurately translate SQL queries from one database system to another, ensuring the syntax and functions are correctly adapted. For example, it can translate a MySQL GROUP_CONCAT function to PostgreSQL's STRING_AGG function or convert MySQL's CURDATE() to PostgreSQL's CURRENT_DATE. This ensures that the queries perform the desired operations in the target database system.

Handling Data Types and Constraints.

Different database systems have unique ways of defining data types and constraints. ChatGPT can help by identifying these differences and providing the correct translations, for instance, converting MySQL's AUTO_INCREMENT to PostgreSQL's SERIAL, or SQL Server's IDENTITY to Oracle's sequence and trigger mechanism. By doing so, ChatGPT helps maintain data integrity and consistency during the migration.

Providing Alternative Solutions

Some proprietary features in one database system may not have direct equivalents in another. ChatGPT can suggest alternative solutions or workarounds to achieve the same functionality in such cases. For example, if a specific function or feature in MySQL does not exist in PostgreSQL, ChatGPT can propose a combination of other functions or custom logic to replicate the behavior.

Guiding Through Complex Transitions

ChatGPT can guide users through complex transitions, highlighting potential changes in query execution or outcomes due to differences in how database systems interpret and handle SQL. This includes differences in transaction handling, error management, indexing, and case sensitivity. By providing insights and recommendations, ChatGPT helps ensure a smoother transition.

Notifying About Potential Differences

Knowing any differences that might affect query results or performance in the target database system is crucial. ChatGPT can notify users of these potential discrepancies and suggest how to adapt queries to ensure consistent results. For example, it can highlight differences in date functions, string concatenation, or conditional logic and make appropriate adjustments.

ChatGPT use cases for SQL-related tasks.

Using ChatGPT for SQL tasks extends beyond simple query translation. Here are several practical use cases where ChatGPT can assist with SQL-related tasks:

10 Ways ChatGPT is Revolutionizing SQL Practices.

It provides an in-depth look at how ChatGPT is successfully used in SQL practices.

Common Pitfalls & Solutions

Pitfall 1: Misinterpretation of Query Intent

Sometimes, ChatGPT may not correctly interpret the intent of the SQL query, leading to incorrect translations between SQL dialects.

Solution:

Be clear and specific when inputting your SQL queries. If you notice a misinterpretation, try rephrasing your query or breaking it down into simpler parts.

Pitfall 2: Unfamiliarity with Database-specific Features.

Some databases have proprietary features that others do not, which can lead to confusion or errors when translating queries.

Solution:

Before migrating to a new database, familiarize yourself with the specific features and syntax of that system. ChatGPT can provide alternative solutions for features that do not have direct equivalents.

Pitfall 3: Overlooking Data Types and Constraints.

Different databases handle data types and constraints differently. Overlooking these differences can lead to data inconsistency.

Solution:

Always verify the translated queries and check for data type and constraint translations. ChatGPT can assist in identifying these differences and providing the correct translations.

Pitfall 4: Ignoring Potential Performance Differences

The performance of a query can vary between different database systems due to differences in how they handle SQL.

Solution:

Be aware of potential performance differences. Use ChatGPT to obtain insights into how different database systems handle SQL and adapt your queries accordingly.
Remember, while ChatGPT is an excellent tool for SQL tasks, it's crucial to double-check the translations and understand the nuances of different database systems.

Future Developments:

Given the dynamic nature of both AI and SQL development, we can expect several advancements:

1. Improved Accuracy: Future versions of ChatGPT are likely to offer even more accurate translations of SQL queries between different database dialects. This will make it easier for developers to switch between different SQL systems.
2. Expanded Database Support: As new database systems and SQL dialects emerge, ChatGPT will likely expand its support to include these new technologies, making it even more versatile.
3. Detailed Explanation of Queries: Future iterations may offer more detailed explanations of complex SQL queries, making it easier for developers to understand and optimize their database interactions.
4. Integration with More Tools: We can anticipate tighter integration with various database management and development tools, providing developers with a more seamless and efficient workflow.
5. Active Learning: Using AI, ChatGPT could learn from its interactions, improving its responses over time and providing even more value to developers.
6. Enhanced Performance Optimizations: With advancements in AI, ChatGPT could provide suggestions for performance optimization in SQL queries, helping developers improve their databases' efficiency and speed.

SQL Companion in GPT Store. Try it now!

Free telegram bot streamlines SQL related tasks

DBConvert Tools for Database Migration and Synchronization

DBConvert offers powerful tools for automating database migration and synchronization across various systems, such as MySQL, PostgreSQL, SQL Server, and Oracle.

DBConvert Studiosimplifies cross-database migration with features like automated schema conversion, data type mapping, and transformation. Its user-friendly interface allows easy setup of source and target connections, scheduled migrations, and thorough data transfer processes.

DBConvert Streams integrates real-time data using Change Data Capture (CDC) technology. It ensures continuous, multidirectional synchronization, ideal for maintaining high data availability and consistency across multiple databases. Both tools provide robust error handling and logging, ensuring reliable and efficient database management.

Conclusion

In the ever-evolving landscape of database management, transitioning between different SQL dialects can be daunting. Each database system, whether MySQL, PostgreSQL, SQL Server, or Oracle, has its unique set of syntax, functions, and constraints. Navigating these differences is crucial for maintaining data integrity and ensuring optimal performance during migrations.

ChatGPT emerges as a powerful ally in this process, offering accurate translations and insightful guidance. By leveraging its capabilities, developers and database administrators can overcome the complexities of SQL dialect variations. From translating queries and handling data types to suggesting alternative solutions and highlighting potential performance differences, ChatGPT provides comprehensive support throughout the migration journey.

title: Your Roadmap to Database Conversion Excellence. published: true date: 2023-09-14 14:46:02 UTC tags: database,mysql,postgres

canonical_url: https://dbconvert.com/blog/a-comprehensive-guide-to-database-conversion/

Introduction:

Section 1: Planning Your Database Migration

Section 2: Preparing for the Migration

Section 3: Performing the Database Conversion

Section 4: Handling Data and Application Code

Section 5: Post-Migration Testing and Verification

Section 6: Monitoring and Maintenance

Section 7: Best Practices and Tips

Conclusion

Introduction:

Database management systems (DBMS) are the backbone of modern applications and are pivotal in storing and managing critical data. As applications evolve and expand, the necessity for migrating from one DBMS to another frequently arises. This comprehensive guide will delve into the intricate database conversion process, addressing common challenges encountered when transitioning between various DBMS platforms. Whether you're moving from MySQL to PostgreSQL, Oracle to SQL Server, or any other combination, this guide aims to provide valuable insights into seamless database migration.

The Significance of Database Migration:

Database migration is a significant undertaking that various factors can drive. It's a strategic move to ensure your data infrastructure aligns with your application's evolving requirements. Common motivations for database migration include:

1. Performance Optimization: Over time, an application's performance requirements can change. Migrating to a different DBMS can provide better scalability and performance capabilities.
2. Feature Set: For example, MySQL and PostgreSQL exhibit unique sets of features. The article "MySQL vs. PostgreSQL" comprehensively analyzes the tangible differences that set MySQL and PostgreSQL apart.
3. Cost Efficiency: Cost considerations often play a role. PostgreSQL is known for its robust open-source community, making it an attractive option for those seeking cost-effective solutions.
4. Vendor Lock-In: Some organizations migrate to open-source alternatives like PostgreSQL from proprietary systems to avoid vendor lock-in.

Popular Databases:

MySQL, PostgreSQL, Oracle and MS SQL Server have earned their popularity in relational databases, each with unique strengths.

MySQL

• Speed and performance
• Ease of use
• Strong community support

Excellent for web applications, content management systems, and data-driven websites. Widely adopted by companies of all sizes.

PostgreSQL

• Custom functions and data types
• Advanced JSON support
• Full-text search
• PostGIS for geospatial data
• Extensive extension ecosystem

Well-suited for complex data modeling, full-text search, geospatial data, and extensive customization.

Oracle

• Scalability
• Security
• Comprehensive feature set

The top choice for large enterprises and organizations requiring a reliable and robust database management system.

Microsoft SQL Server

• Seamless integration with Microsoft's ecosystem
• Ideal for businesses invested in Microsoft technologies
• Support for critical data and enterprise-level applications

Microsoft Access

• Simple desktop database solution
• Good for small-scale projects
• Easy to learn and use

Commonly used for small businesses and individuals for lightweight database applications.

SQLite

• Embedded, serverless database
• Minimal administration required
• Lightweight and fast

Ideal for mobile apps, embedded systems, and situations where a small, self-contained database is needed.

Firebird

• Open-source relational database
• High performance and scalability
• Supports stored procedures and triggers

Suitable for applications requiring high-performance data management and support for complex business logic.

The Need for Migration:

While both MySQL and PostgreSQL have their strengths, there are scenarios where migrating between them becomes essential. For example:

• Application Growth: As your application grows and demands more complex features, you might find PostgreSQL's advanced capabilities more suitable.
• Data Compatibility: If your data schema evolves or requires better compatibility with specific data types, migrating to PostgreSQL can be advantageous.
• Vendor or Licensing Changes: MySQL licensing or support changes might prompt migrating to an open-source alternative like PostgreSQL.
• Performance Scaling: If your application experiences increased traffic and demands better performance scaling, PostgreSQL might offer a more robust solution.

Reasons for PostgreSQL to MySQL Migration:

• Application Compatibility: Transitioning to a MySQL-centric application ecosystem ensures seamless compatibility and minimizes potential issues.
• Feature Alignment: MySQL offers specific features and extensions that better suit your application's needs, especially if reliant on MySQL-specific functions or plugins.
• Ecosystem Standardization: Aligning with an existing MySQL-based database ecosystem simplifies management and maintenance.
• Performance Optimization: MySQL's performance characteristics may better match your application's workload and query patterns, delivering improved query response times and resource usage.
• Legacy Integration: Facilitate integration with legacy systems or third-party applications requiring MySQL compatibility.

Integration with Microsoft SQL Server (MSSQL) and Oracle:

• Microsoft SQL Server (MSSQL): MSSQL is an ideal choice for organizations deeply integrated into Microsoft's ecosystem. Its compatibility with Microsoft technologies, including Windows Server and Azure, ensures a cohesive environment for managing and analyzing data. Migrating to MSSQL can strengthen the alignment with Microsoft's suite of tools and services.
• Oracle: Migrating to Oracle may be warranted when your organization seeks the utmost data reliability and advanced database capabilities.

These considerations underscore the importance of evaluating your specific database requirements and ecosystem when making migration decisions. Whether moving from PostgreSQL to MySQL, integrating with MSSQL, or adopting Oracle, each choice should align with your application's needs and strategic objectives.

As we embark on this guide, we will delve into the intricacies of planning, executing, and optimizing a successful database migration between different database management systems (DBMS). Whether you're a developer, a database administrator, or an IT decision-maker, this guide will equip you with the knowledge and best practices to navigate the complex terrain of database conversion among these powerful DBMSs.

Section 1: Planning Your Database Migration

Database migration is a complex endeavor that demands careful planning and execution. This section will explore the critical aspects of planning your database migration jobs.

Importance of Planning and Assessing Your Current Database:

• Understanding Your Current State: Before embarking on a migration, thoroughly understanding your existing database is crucial. It encompasses analyzing the schema, data types, indexes, and relationships.
• Data Volume and Complexity: Evaluate the volume and complexity of your data. Large datasets or intricate relationships may impact the migration process and require special attention.
• Data Integrity and Quality: Assess the integrity and quality of your data. Identify any inconsistencies, anomalies, or data quality issues that must be addressed during migration.
• Dependencies and Applications: Identify all dependencies on your database, including applications, scripts, and reporting tools. Understand how the migration will impact these dependencies.
• Performance Metrics: Collect performance metrics for your current database. This data will serve as a baseline for evaluating the performance of the migrated database.

Creating a Migration Strategy and Setting Clear Goals:

• Define Clear Objectives: Clearly articulate the objectives of your migration. Are you seeking improved performance, cost savings, or better compatibility with specific features?
• Choose the Right Migration Path: Decide whether you will perform a homogeneous migration (e.g., MySQL to MySQL) or a heterogeneous migration (e.g., Oracle to PostgreSQL). Your choice will impact the tools and processes you use.
• Select Migration Tools: Research and select the appropriate migration tools and scripts that align with your migration strategy. These tools can help automate various aspects of the migration process.
• Data Mapping and Transformation: Define a data mapping and transformation plan to handle differences in data types, constraints, and schema structures between various database types.
• Testing and Validation: Develop a comprehensive testing plan that includes unit testing, integration testing, and data validation. Ensure that the migrated database meets your performance and functionality goals.
• Rollback Plan: Prepare a rollback plan if unexpected issues arise during migration. This plan should outline steps to revert to the previous state without data loss.

Potential Challenges and How to Overcome Them:

Data Type and Schema Differences

MySQL, PostgreSQL, MSSQL have different data types and schema structures. Mapping and transforming data can be complex.

Solution:

• Define a precise data mapping and transformation strategy.
• Utilize available tools for schema conversion.

Application Compatibility

Applications may have queries or code specific to the DBMS. Compatibility issues can arise.

Solution:

• Modify and test application code to ensure compatibility with the target DBMS.

Data Volume and Downtime

Large datasets and long migration times can result in significant downtime.

Solution:

• Implement strategies such as data chunking and parallel processing to minimize downtime.

Data Validation

Ensuring data integrity and accuracy during migration is essential.

Solution:

• Develop comprehensive data validation scripts and conduct thorough testing.

Performance Optimization

The performance of the migrated database may need to meet expectations.

Solution:

• Continuously monitor and optimize the performance of the new database post-migration.

By meticulously planning your database migration, defining clear goals, and understanding the potential challenges, you set the foundation for a successful transition. The following sections will dive into the practical steps in executing the migration, including data mapping, code adaptation, and post-migration testing.

Section 2: Preparing for the Migration

A well-prepared foundation is essential for a smooth database migration. This section will explore the prerequisites, backup procedures, and the significance of testing in a controlled environment.

Prerequisites for the Migration Process:

• Database Servers: Ensure your source and target database servers are correctly set up, configured, and accessible.
• Migration Tools: Acquire and install the tools and scripts that align with your migration strategy. Popular tools include DBConvert Studio, pgloader, SQL Server Migration Assistant (SSMA) for MSSQL, Oracle SQL Developer for Oracle databases, and MySQL Workbench for schema conversion.

DBConvert Studio offers the flexibility to convert and synchronize data seamlessly in any combination between different database systems.

• Access Credentials: Have the appropriate access credentials for the source and target databases, including usernames and passwords with sufficient privileges.
• Network Connectivity: Confirm that network connectivity between the source and target databases is stable and secure. Ensure that firewalls and security groups allow the necessary traffic.
• Target Schema: Prepare the target schema structure in your target database. Create tables, indexes, constraints, and sequences to match your source database schema. Tools like DBConvert Studio can generate an equivalent schema on the destination database mirroring the source schema.

How to Back Up Your Existing Database:

1. Full Database Backup:

Perform a complete source database backup using tools or database management interfaces. This backup ensures that your data is safe in case of unexpected issues during migration.

To backup a MySQL database, you can use the mysqldump command:

mysqldump -u username -p dbname > backup.sql

For Oracle databases, you can use the expdp (Data Pump Export) utility:

expdp username/password@tns_alias dumpfile=backup.dmp

To backup a PostgreSQL database, you can use the pg_dump command:

pg_dump -U username dbname > backup.sql

For SQL Server databases, you can use the sqlcmd utility to generate a backup script:

sqlcmd -S servername -d dbname -U username -P password -Q "BACKUP DATABASE dbname TO DISK='backup.bak'"

2. Backup Verification:

After creating the backup, verify its integrity by restoring it to a test environment. Ensure that the restored database is consistent and error-free.

3. Data Consistency During Migration:

Plan to perform a final backup before the migration process begins to capture any changes made to the database during the migration planning phase.

The Importance of Testing in a Controlled Environment:

• Create a Sandbox Environment: Set up a controlled testing environment that mirrors your production environment as closely as possible. This environment should include the target database and any dependent applications.
• Migration Dry Run: Conduct a migration dry run in the sandbox environment. Use a subset of your data to simulate the migration process without affecting the production database. This step allows you to identify and address potential issues before the migration.
• Performance Testing: Assess the performance of the migrated database in the controlled environment. It includes running performance benchmarks and ensuring the database meets your performance expectations.
• Functional Testing: Test all application functionality against the migrated database. Ensure that queries, stored procedures, and application logic work as expected.
• Data Validation: Implement data validation scripts to confirm the accuracy and integrity of the data after migration. Verify that data transformations and mappings are correct.
• User Acceptance Testing (UAT): If applicable, involve end-users in user acceptance testing. Gather feedback and ensure that the migrated system meets user requirements and expectations.
• Rollback Testing: Develop and test your rollback plan in a controlled environment to ensure you can revert to the previous state if necessary.

Testing in a controlled environment is paramount to minimize risks and ensure a successful migration. Any issues identified during testing can be addressed before migrating your production database. The following section will explore the migration process, including data mapping, transformation, and code adaptation.

Section 3: Performing the Database Conversion

Performing a database conversion involves several steps that require careful consideration and execution. This section will guide you through a step-by-step process, provide code examples and scripts, and address data mapping, data type handling, and schema differences.

Step-by-Step Process of Database Migration:

1. Schema Assessment: Review the schema of your source database and target database. Identify differences in data types, constraints, and schema structures.
2. Data Mapping and Transformation: Create a comprehensive plan for data mapping and transformation. Determine how data from the source db will be mapped to target, considering differences in data types and constraints.
3. Schema Conversion: Use migration tools or scripts to convert the schema from the source to destination database syntax. Here's a basic example for converting a table:

-- PostgreSQL
CREATE TABLE "products" (
"id" SERIAL PRIMARY KEY,
"name" VARCHAR(255),
"price" NUMERIC(10, 2)
);

PostgreSQL create table syntax

/* MySQL */
CREATE TABLE `products` (
`id` INT AUTO_INCREMENT PRIMARY KEY,
`name` VARCHAR(255),
`price` DECIMAL(10, 2)
);

MySQL create table syntax

/* Oracle */
CREATE TABLE products (
  id NUMBER GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
  name VARCHAR2(255),
  price NUMBER(10, 2)
);

Oracle create table syntax

-- MS SQL Server
CREATE TABLE products (
  id INT IDENTITY(1,1) PRIMARY KEY,
  name VARCHAR(255),
  price NUMERIC(10, 2)
);

SQL Server create table syntax

Schema Conversion with DBConvert Studio: If you prefer an automated approach and want to simplify the schema conversion process, consider using DBConvert Studio. This tool can automatically convert your database schema from the source to the destination database syntax, eliminating the need for manual scripting.

4. Data Migration: Use specialized tools or scripts to migrate. Ensure that data transformations and mappings are correctly applied.

5. Indexes and Constraints: Recreate indexes and constraints in the target database that existed in source.

DBConvert Studio provides an intuitive user interface that allows you to map data types, set constraints, and perform schema transformations with ease. This is especially beneficial for users who may not have extensive scripting experience, streamlining the migration process and reducing the risk of errors.

6. Stored Procedures and Functions: If your application relies on stored procedures or functions, rewrite them in the target's syntax. The logic may need to be adjusted to match the target db procedural language.

7. Views and Triggers: Recreate views and triggers in target db, ensuring they function as intended.

8. Application Code Updates: Modify your application code to adapt to target-specific SQL syntax and features.

Data Mapping, Data Type Handling, and Schema Differences:

1. Data Mapping: When performing a database migration, it's crucial to map your data accurately from the source to the target, considering the differences in data types between the database systems.
2. Schema Differences: Be aware of schema differences. PostgreSQL is stricter with data type casting and enforces more rigorous SQL standards compliance than MySQL.
3. Case Sensitivity: PostgreSQL and Oracle are case-sensitive by default, while MySQL and SQL Server are usually case-sensitive. Ensure that your queries and table/column references match the case sensitivity rules of PostgreSQL.
4. Serial Columns: In PostgreSQL, you often use SERIAL data types for auto-incrementing columns instead of MySQL's AUTO_INCREMENT. In MS SQL Server, you commonly use the IDENTITY property to create auto-increment columns. In Oracle, you typically use a SEQUENCE and a trigger to achieve auto-increment behavior for a column.
5. Character Encoding: Pay attention to character encoding differences between databases and ensure your data is converted correctly.
6. Primary Keys and Unique Constraints: Review primary keys and unique constraints, as they might need to be modified to adhere to target's requirements.

Migrating databases involves careful attention to detail, as differences between databases can impact data integrity and application functionality. Using migration tools and scripts specific to the task can simplify the process. Additionally, thorough testing and validation are essential to ensure a successful migration. The next section will explore the critical aspects of testing and validating the migrated database.

Section 4: Handling Data and Application Code

Migrating a database involves more than just transferring data; it also entails adapting the data and application code to ensure compatibility with the target database system. In this section, we'll discuss the crucial tasks of transforming data and adjusting application code during migration.

Transforming and Adapting Data During Migration:

• Data Mapping and Transformation: As discussed earlier, define a clear data mapping and transformation plan to handle differences in data types, constraints, and schema structures between databases. Use migration tools or scripts to automate data transformation whenever possible.
• Data Cleansing and Validation: Before migrating data, perform data cleansing and validation to address data quality issues, anomalies, or inconsistencies. Ensure that data integrity is maintained throughout the migration process.
• Null Handling: Be mindful of how NULL values are handled in your data. PostgreSQL, MSSQL and Oracle may have different default behaviors for NULL values compared to MySQL. In MySQL, NULL is a special marker used to indicate that a data value does not exist in the database. It represents the absence of a value or an unknown value. Adjust your data accordingly.
• Date and Time Conversions: When migrating date and time values between DBMSs like Oracle, MSSQL, MySQL, and PostgreSQL, it's essential to understand the date types, formats, and time zone handling specific to each system. You may need to use conversion functions and update time zone data to ensure accurate date and time representations in the target database.

Adjusting Application Code and Queries for Compatibility:

• SQL Syntax Differences: Review your application's SQL queries and adapt them to target db syntax. Pay attention to differences in date functions, string concatenation, and mathematical operations.
• Quoting Identifiers: PostgreSQL and Oracle uses double quotes (") for quoting identifiers (table and column names with spaces or special characters), while MySQL uses backticks. Adjust your queries accordingly. In MSSQL, identifiers are typically enclosed in square brackets ([]) or (") for quoting.
• Stored Procedures and Functions: To ensure a smooth database migration, rewrite stored procedures, functions, and triggers to match the procedural language of the target DBMS. Common languages include:
• MySQL: SQL/PSM
• MSSQL: T-SQL
• PostgreSQL: PL/pgSQL (with support for other languages)

• Oracle: PL/SQL

• Database Connection Parameters: Update your application's database connection parameters to connect to the target database. Adjust the host, port, username, and password as needed.

• Data Retrieval Methods: Modify data retrieval methods in your application code to handle differences in how data is returned by DBMS.
• Testing and Validation: Rigorously test your application after making code adjustments to ensure it functions correctly with the migrated database.
• User Training: Train users and developers on any changes introduced by the migration, including differences in SQL syntax and database behavior.
• Documentation Updates: Update your application's documentation to reflect any changes in database interactions and queries due to the migration.
• Backward Compatibility: Consider maintaining backward compatibility with the original database system during the transition period, especially if the migration is part of a phased approach.

Adapting your application code and queries is critical to a successful database migration. It ensures that your application continues to function as expected and takes full advantage of the capabilities of the target database system. Thorough testing and validation of the application are essential to catch any compatibility issues early in the migration process. The next section will explore the importance of post-migration testing and verification.

Section 5: Post-Migration Testing and Verification

The work doesn't end once the data and application code have been migrated. Dedicating time and resources to thorough post-migration testing and verification is crucial. In this section, we'll emphasize the importance of these steps and discuss how to ensure data integrity and functionality while handling unforeseen issues.

The Importance of Thorough Testing Post-Migration:

• Data Integrity Assurance: Post-migration testing is essential to verify that data has been accurately and completely migrated from the source database to the target database. Confirm that primary keys, foreign keys, and constraints are functioning as expected.
• Application Functionality Validation: Ensure that all aspects of your application, including user interfaces, reports, and backend processes, work as expected with the migrated database. Test all application functions and features to guarantee they continue performing as intended.
• Performance Evaluation: Benchmark the performance of the migrated database against predefined criteria and expectations: measure query response times and system resource utilization. Identify and address any performance bottlenecks or issues.
• Data Validation: Implement comprehensive data validation scripts to confirm the accuracy and integrity of data in the migrated database. Verify that data transformations and mappings have been executed correctly.
• User Acceptance Testing (UAT): Involve end-users and stakeholders in user acceptance testing. Gather feedback on the functionality and usability of the application with the new database.

Verifying Data Integrity and Functionality:

• Data Validation Queries: Develop SQL queries and scripts to validate data integrity. Check for data anomalies, missing records, and inconsistencies.
• Regression Testing: Perform regression testing on your application to ensure the migration hasn't compromised existing functionality. Re-run previously successful test cases to verify that they still pass.
• Boundary Testing: Test boundary conditions and edge cases to ensure that the migrated system can handle extreme scenarios without issues.
• Security and Access Control: Validate that the target database has correctly implemented security measures and access controls. Ensure that only authorized users can access sensitive data.
• Load and Stress Testing: Conduct load and stress testing to assess how the system behaves under heavy loads. Identify performance bottlenecks and scalability issues.
• Backup and Recovery Testing: Test backup and recovery procedures to ensure data can be restored in case of unexpected issues. Verify that the rollback plan is functional.

Handling Unforeseen Issues:

• Issue Identification: Be prepared to identify and document any unforeseen issues during post-migration testing—track issues related to data, functionality, and performance.
• Issue Prioritization: Prioritize issues based on their impact on data integrity, functionality, and business operations. Address critical issues promptly.
• Root Cause Analysis: Conduct a thorough root cause analysis for critical issues to understand the underlying reasons. This analysis can inform corrective actions.
• Issue Resolution: Collaborate with the database migration team, developers, and stakeholders to resolve identified issues. Implement fixes, code changes, or adjustments as needed.
• Communication and Documentation: Maintain clear communication with stakeholders throughout the issue resolution process. Document the steps taken to address issues and their outcomes.

Post-migration testing is a critical phase that ensures a successful transition to the new database system. It helps maintain data integrity, assures application functionality, and prepares your organization for unexpected challenges. Be vigilant, organized, and responsive to issues that may arise, and remember that post-migration testing is an ongoing process that continues even after the migration is complete. The following section will explore best practices and tips for a successful database conversion.

##

Section 6: Monitoring and Maintenance

The section of our guide focuses on the critical aspects of ongoing monitoring and maintenance for your database system post-migration. We'll discuss the importance of these activities, highlight tools and techniques for performance monitoring, and provide guidance on troubleshooting common issues.

Ongoing Monitoring and Maintenance Post-Migration:

1. Continuous Monitoring: Establish a robust monitoring system to keep a close eye on the health and performance of your migrated database. Regularly review system metrics, logs, and alerts.
2. Database Backups: Maintain a reliable backup strategy, including regular full and incremental backups. Test the restoration process periodically to ensure data recoverability.
3. Routine Maintenance Tasks: Schedule everyday maintenance tasks, such as vacuuming, reindexing, and compaction, to optimize database performance and prevent fragmentation.
4. Security Audits: Conduct regular security audits and vulnerability assessments to identify and address potential security risks and vulnerabilities.
5. Performance Tuning: Continuously fine-tune your database for optimal performance. Analyze query execution plans, optimize indexes, and adjust configuration settings.

Tools and Techniques for Performance Monitoring:

1. Database Management Systems (DBMS) Tools: Most DBMSs offer built-in performance monitoring and optimization tools. Familiarize yourself with the tools provided by your specific DBMS.
2. Third-Party Monitoring Tools: Consider using third-party monitoring tools such as Prometheus or Grafana for advanced performance monitoring, alerting, and visualization.
3. Query Profiling: Use query profiling tools to analyze and optimize slow or resource-intensive queries. Identify bottlenecks and make necessary adjustments.
4. Log Analysis: Regularly review database logs for error messages, warnings, and performance-related information. Log aggregation tools like ELK Stack can be invaluable.

Troubleshooting Common Issues:

1. Performance Degradation: If you notice performance degradation, investigate the root cause by analyzing query performance and system metrics. Adjust configurations, optimize queries, and consider hardware upgrades if necessary.
2. Concurrency Issues: Monitor database locks and concurrent connections. Address contention issues by optimizing transactions and locking strategies.
3. Data Corruption: Implement integrity checks and regularly validate data to promptly detect and rectify data corruption issues.
4. Security Breaches: In case of a security breach or data breach, follow your incident response plan. Investigate the breach, implement security patches, and enhance security measures.
5. Resource Exhaustion: Monitor resource utilization, including CPU, memory, and disk space. Implement resource scaling or optimization strategies to prevent resource exhaustion.
6. Backup Failures: When backup failures occur, identify the cause and resolve it promptly. Regularly test backup and recovery procedures to ensure data recoverability.
7. Software Updates: Stay updated with software updates and security patches for your DBMS, operating system, and third-party tools. Apply updates in a controlled manner to minimize disruption.
8. Documentation and Knowledge Sharing: Maintain detailed documentation of your monitoring and maintenance activities. Share knowledge and best practices with your team to build a collective understanding of the database system.

By prioritizing ongoing monitoring and maintenance, you can ensure your migrated database system's long-term stability, performance, and security. Troubleshooting common issues as they arise and proactively addressing potential problems will help you maintain a robust and reliable database environment.

Section 7: Best Practices and Tips

In the final section of our guide, we'll delve into proven best practices, offer tips to avoid common migration pitfalls, and discuss strategies for optimizing the performance of your newly migrated database.

Proven Best Practices for a Successful Database Conversion:

1. Thorough Planning: Invest time in comprehensive planning and assessment of your current database and migration goals.
2. Data Mapping and Transformation: Develop a well-defined data mapping and transformation plan to handle schema and data differences between source and target databases.
3. Testing: Rigorously test and validate your migration in a controlled environment before migrating the production database.
4. Data Validation: Implement robust data validation to ensure data integrity during and after migration.
5. Backup and Rollback Plans: Maintain reliable backup and rollback plans to safeguard against unexpected issues.
6. Application Code Adaptation: Modify application code and queries to align with the syntax and features of the target database.
7. Post-Migration Testing: Dedicate time and resources to comprehensive post-migration testing to validate data, functionality, and performance.

Tips for Avoiding Common Migration Pitfalls:

1. Engage Experts: Consider involving database experts or consultants with database migrations experience.
2. Incremental Migration: Perform incremental or phased migrations to minimize downtime and reduce risks.
3. Documentation: Maintain detailed documentation throughout the migration process, including decisions, configurations, and issue resolutions.
4. Monitoring: Implement database performance monitoring and alerting to identify and address post-migration issues promptly.
5. User Training: Train end-users and developers on the new database system and any changes in SQL syntax.
6. Performance Benchmarking: Continuously monitor and benchmark the performance of the migrated database to identify areas for optimization.
7. Regular Updates: Keep your database system and migration tools updated with the latest releases and patches.

Database Optimization for Performance:

1. Indexing: Ensure appropriate indexes are in place for frequently queried columns to optimize query performance.
2. Query Optimization: Continuously analyze and optimize SQL queries to improve efficiency and reduce query execution times.
3. Caching: Implement caching mechanisms to reduce the load on the database system, especially for frequently accessed data.
4. Partitioning: Consider database table partitioning to improve query performance on large datasets.
5. Regular Maintenance: Schedule routine database maintenance tasks such as vacuuming and reindexing to keep the database running efficiently.
6. Database Scaling: Be prepared to scale your database horizontally or vertically to handle increased workloads as your application grows.
7. Query Monitoring: Use query monitoring tools to identify slow or inefficient queries and address them promptly.
8. Security: Continuously assess and enhance the security of your database system to protect against potential threats.

By adhering to best practices, staying vigilant for common pitfalls, and implementing performance optimization strategies, you can ensure a successful database conversion that meets your objectives and provides a robust foundation for your application. Remember that a well-executed migration can improve performance, reduce costs, and enhance functionality, ultimately benefiting your organization and its users.

Section 9: Conclusion

In this comprehensive guide to database conversion between different database types in heterogeneous environments, we've explored the intricate process of migrating your database from one system to another. Here, we summarize key takeaways, highlight the benefits of a well-executed migration, and encourage readers to seek professional assistance.

Key Takeaways:

• Planning is Paramount: Thoroughly plan your migration, assess your current database, and set clear objectives. A well-structured plan is the foundation of a successful migration.
• Testing is Crucial: Rigorous testing, both before and after migration, is essential to ensure data integrity, application functionality, and performance.
• Data Mapping and Transformation: Pay close attention to data mapping and transformation to address differences in data types, constraints, and schema structures.
• Application Code Adaptation: Modify application code and queries to align with the syntax and features of the target database system.
• Performance Monitoring and Maintenance: Implement ongoing monitoring and maintenance practices to optimize performance, ensure security, and troubleshoot common issues.
• Documentation and Training: Maintain comprehensive documentation throughout the migration process and provide training to users and developers regarding the new database system.

Benefits of a Well-Executed Database Migration:

• Improved Performance: A successful migration can improve database performance, allowing your applications to scale more effectively.
• Cost Savings: You can reduce licensing and maintenance costs by migrating to an open-source database system.
• Advanced Features: Take advantage of advanced features and capabilities offered by the target database system, enhancing the functionality of your applications.
• Data Integrity: Ensure data integrity throughout migration, preventing data loss or corruption.
• Reduced Vendor Lock-In: Migrating to an open-source system reduces dependency on a single vendor, offering more control and flexibility.
• Enhanced Security: Implement security best practices in the target database system to safeguard your data and applications.

Seek Professional Assistance:

Database migration can be complex, and challenges may arise that require expert guidance. If you encounter obstacles or are unsure about specific aspects of your migration, feel free to seek professional assistance. Database experts and consultants with experience in migrations can provide valuable insights and solutions to ensure a smooth transition.

In closing, database migration is a strategic move that, when executed thoughtfully, can bring about significant benefits for your organization. It's a process that requires meticulous planning, thorough testing, and ongoing maintenance. By following the best practices outlined in this guide and staying vigilant, you can embark on a successful journey to migrate your databases, setting the stage for a more efficient and capable database environment. Please share your experiences, learn from others, and don't hesitate to seek our expert assistance to make the most of your database migration endeavor.

Discover DBConvert Studio – Your Database Conversion and Sync Solution!

🌐 Wide Compatibility: Support for MySQL, PostgreSQL, SQL Server, Oracle, and more.

🔄 Multi-Directional: Convert and sync data in any direction with ease.

🛠️ User-Friendly: No coding needed, an intuitive interface for all levels.

🔒 Data Integrity: Keep data accurate and consistent during migration.

📈 High-Performance: Swift, efficient data transfers for reduced downtime.

📊 Visual Mapping: Easy source-to-target mapping visualization.

Use DBConvert Studio and simplify database migrations now!

Download DBConvert Studio

title: Microsoft Access: Unveiling the Dual Power of Database and Front-End Development published: true date: 2023-06-03 15:06:44 UTC tags: access,database,msaccess

canonical_url: https://dbconvert.com/blog/access-frontend-backend/

In the realm of software development, various tools and technologies aid in creating robust and efficient solutions. One such tool that has been empowering developers for decades is Microsoft Access. In this article, we will delve into the capabilities of Microsoft Access and explore how it enables developers to harness the dual power of database management and front-end development.

Introduction:

Microsoft Access often remains underestimated, considered merely a toy database by many. However, it is time to shed light on its true potential. Access is a unique product that combines a database engine with a powerful front-end development tool called Visual Basic for Applications (VBA). In this article, we will explore the capabilities of both aspects and debunk the myths surrounding Access.

Database Management: The Backbone of Access.

At its core, Microsoft Access is a relational database management system (RDBMS) that allows users to store, organize, and manipulate large amounts of data. With Access, developers can create tables, define relationships between them, and establish data integrity constraints.

This powerful database engine enables efficient querying and retrieval of data, making it a valuable asset for managing vast amounts of information.

Access provides a user-friendly interface for designing and creating database schemas, making it accessible to developers of varying expertise levels. With its intuitive table design view, developers can easily define fields, data types, and primary key constraints.

Access also includes a query builder that allows for creating complex queries without extensive SQL knowledge. These features streamline the database development process and empower developers to build robust data structures quickly.

Access database limitations:

Access database is ideal for small to mid-sized applications. While it supports multiple users, it is not recommended for heavy concurrent usage. Here are some theoretical limitations to consider:

• Maximum database size : 2 gigabytes, but can be bypassed by linking to tables in other Access databases.

• Total number of objects : 32,768 objects.

• Number of modules : 1,000 modules, including forms and reports.

• Character limits : Object names (64 characters), passwords (14 characters), and user/group names (20 characters).

• Maximum concurrent users : 255 users (theoretical).

However, the practicality of some of these limitations may vary.

Front-End Development: Bringing Data to Life.

While Microsoft Access excels in database management, it also offers robust front-end development capabilities.

Access includes an effective forms designer that enables developers to create user interfaces to interact with the underlying data. The forms designer provides various controls, such as text boxes, buttons, and dropdown menus, allowing developers to create intuitive and user-friendly interfaces.

The forms designer in Access supports various data binding options, allowing developers to link form controls directly to the database. This way, changes made on the form are automatically reflected in the database and vice versa, eliminating the need for manual synchronization.

Integration and Extensibility: Going Beyond the Basics

Microsoft Access seamlessly integrates with other Microsoft Office applications, such as Excel, Word, and Outlook. This integration allows developers to leverage the power of Access in conjunction with other tools to create comprehensive solutions. For example, data from Access can be exported to Excel for further analysis or presented in Word documents for reporting purposes. By combining the strengths of different Microsoft Office applications, developers can create cohesive and interconnected solutions that cater to various business needs.

Furthermore, Microsoft Access supports external data sources, enabling developers to connect to and interact with data from other systems and databases. This capability opens up possibilities for hybrid solutions leveraging data from multiple sources. Whether connecting to an SQL Server database or consuming data from a web service, Access provides the tools and connectors to integrate disparate data sources effortlessly.

DBConvert Conversion and Synchronization Capabilities:

DBConvert's conversion and synchronization capabilities are instrumental in enhancing the functionality of Microsoft Access and expanding its capabilities by migrating/ synching data with MS SQL Server, MySQL, PostgreSQL, Oracle, and more.

With DBConvert, users can overcome the limitations of Access and seamlessly migrate their data to another database platforms.

Additionally, DBConvert enables bidirectional synchronization between Access and other RDBMS, ensuring that changes made in one database are reflected in the other, maintaining data consistency across the hybrid system.

DBConvert simplifies transferring Access data to popular cloud databases like Amazon RDS, Microsoft Azure SQL Database, and Google Cloud SQL. Moving data from MS Access opens up new possibilities for storing and managing data in the cloud, providing increased accessibility, scalability, and flexibility.

Summary:

Access is often misunderstood as a toy database, yet it has immense potential. While it caters to smaller databases, elite Access developers utilize it as a front-end targeting "serious" databases like PostgreSQL, SQL Server, and Oracle. Microsoft Access is a testament to the power of combining database management and front-end development in a single tool. Access can seamlessly connect to various back-end databases with minimal code changes, offering developers a significant advantage.

It's time to recognize the true power of Microsoft Access beyond its perceived limitations. So, embrace Access as a valuable database and front-end development tool, and unlock a world of possibilities.

If you find yourself constrained by the limitations of Access, DBConvert offers a pathway to unleash the full potential of your data. With its robust conversion and synchronization capabilities, DBConvert empowers users to optimize their data management systems and achieve enhanced efficiency and flexibility and take your database operations to the next level.

Migrate to Access? Check out DBConvert Converters and Synchronization software.

title: Simplifying Data Migration: Converting DBF Files to SQL Server. published: true date: 2023-05-19 12:06:43 UTC tags: sqlserver,foxpro,database,mssql

canonical_url: https://dbconvert.com/blog/converting-dbf-to-sql-server/

Introduction.

DBF (dBase File) is a widely recognized format for storing structured data. It has gained significant popularity among applications, particularly those operating on Microsoft DOS and Windows platforms. Initially, DBF files were closely associated with the dBase database management system, which enjoyed widespread use in the 1980s and 1990s. Over time, other database management systems like FoxPro, Clipper, and Visual FoxPro adopted the DBF format.

In the ever-evolving world of data management, businesses often face the challenge of migrating their data from legacy systems to more modern and scalable platforms. One such migration scenario involves converting DBF to SQL Server, commonly associated with dBase, XBase, Clipper, FoxBase and FoxPro databases.

This article explores the process of converting DBF to SQL Server, highlighting the benefits and providing insights into best practices for a robust transition.

Understanding the Need for Migration.

As technology progresses, legacy systems like FoxPro may no longer meet the demands of modern businesses. On the other hand, SQL Server offers a robust and feature-rich environment that can handle large volumes of data, provides enhanced security measures, and supports high-performance transactional applications. Migrating from DBF files to SQL Server ensures longevity, scalability, and compatibility with current and future data management needs.

Benefits of Migrating to SQL Server.

1. Scalability: SQL Server offers scalability options to accommodate growing data volumes and user demands, ensuring your system can handle future growth.

2. Performance: SQL Server's optimized query processing and indexing capabilities enable faster and more efficient data retrieval, enhancing application performance.

3. Security: SQL Server provides robust security features, including encryption, user access controls, and data masking, protecting your valuable data from unauthorized access.

4. Integration: SQL Server smoothly integrates with other Microsoft products, facilitating integration with business intelligence tools, reporting services, and analytical platforms.

Choosing the Right Conversion Tool.

Selecting a reliable and efficient conversion tool is crucial to simplify the DBF to SQL Server conversion process. DBConvert/ DBSync for FoxPro and MSSQL software streamlines the migration process, providing a powerful and user-friendly solution. It offers comprehensive features, including intuitive user interfaces and efficient data mapping capabilities.

Step-by-Step Conversion Process.

1. Assess Your DBF Files: Begin by analyzing the structure and content of your DBF files. Understand the relationships, data types, and any specific requirements for the migration process.

2. Prepare the SQL Server Database: Create a new database or identify an existing one to serve as the target for your migrated data. Ensure that the SQL Server instance is configured correctly and accessible.

3. Install and Configure the Data Migration Tool Install the appropriate software, either DBConvert or DBSync, based on your specific task, whether conversion or synchronization. Proceed to configure the basic settings, including establishing connections for the source and target databases.

4. Map the Data In this crucial step, you must define the mapping between the fields in your DBF files and the corresponding columns in the SQL Server table. It is essential to consider data types, indices, and constraints to ensure a precise and efficient data transfer.

5. Execute the Conversion: Start the conversion process and monitor its progress. The tool provides real-time feedback, allowing you to track potential errors or issues requiring attention.

6. Verify and Validate: After the conversion, thoroughly verify the migrated data in the SQL Server database. Perform data integrity checks, validate relationships, and ensure data is accurately represented.

If you're using the DBConvert software, you'll be pleased to know that it automates the mapping process, making it a seamless experience. The software intelligently analyzes the structures of the DBF files and SQL Server tables and automatically establishes the mappings. However, if you desire further customization, the software also allows you to adjust the mappings according to your specific requirements manually. This flexibility will enable you to tailor the data mapping to meet your unique needs, ensuring a highly personalized data migration process.

Synchronization.

Synchronizing data between legacy DBF files and SQL Server databases can be challenging due to the differences in their underlying technologies and structures. However, DBSync tool from DBConvert provides a reliable solution to streamline the synchronization process and ensure data consistency across these disparate systems.

Legacy FoxPro databases often serve as the backbone of critical applications in many organizations. However, as businesses evolve, there is a growing need to integrate these legacy systems with modern SQL Server databases to leverage advanced features, scalability, and improved data management capabilities.

DBSync handles the complexities of data mapping and transformation between FoxPro and SQL Server. It intelligently detects and resolves conflicts that may arise during the synchronization process, ensuring data integrity and minimizing the risk of data inconsistencies. This intelligent synchronizer mechanism considers each database platform's specific characteristics and provides accurate data replication.

With DBSync, organizations can schedule automated synchronization tasks to run at predefined intervals. This feature eliminates manual intervention and ensures data remains synchronized without constant monitoring. Additionally, the software provides detailed logs and reports to track the synchronization process, allowing for easy troubleshooting and auditing.

The user-friendly interface of DBSync simplifies the configuration and management of synchronization tasks. It guides users through the process with a wizard-like interface, making it easy to set up connections to both the legacy DBF Table and the SQL Server database. Users can define the mapping between tables, specify synchronization options, and customize the synchronization process to meet their requirements.

Conclusion:

Migrating DBF files to SQL Server allows businesses to modernize their data management systems, unlock advanced features, and ensure long-term scalability. By following a well-defined conversion process and utilizing reliable tools like DBConvert & DBSync, businesses can seamlessly migrate their data from DBF files to the robust SQL Server platform. Embracing this migration not only future-proofs data management practices but also enables organizations to leverage the full potential of their data in an ever-evolving digital landscape.

title: MySQL vs PostgreSQL in 2023. published: true date: 2023-02-01 11:00:06 UTC tags: database,postgresql,mysql,postgres

canonical_url: https://dbconvert.com/blog/mysql-vs-postgresql/

Regarding choosing a relational database management system (RDBMS), two popular options are PostgreSQL and MySQL. Both have been around for decades and have proven highly reliable, secure, and scalable. However, they have different strengths and weaknesses that make one more suitable for specific use cases. This article will compare PostgreSQL and MySQL to help you make an informed decision in 2023.

History and Development

PostgreSQL was first released in 1996 and has become a widely used open-source RDBMS. It is known for its strong adherence to SQL standards, robust feature set, and focus on data integrity and security.

MySQL, on the other hand, was first released in 1995 and has been widely used for web-based applications due to its high performance and ease of use. Over time, Oracle acquired the open-source database management system MySQL and turned it into a commercial product.

Features

PostgreSQL and MySQL both offer a wide range of features as relational database management systems, but there are some key differences between the two:

1. Data Types: PostgreSQL supports a broader range of advanced data types, including arrays, hstore (a key-value store), and JSONB (binary JSON), which provide more flexible and efficient data storage options. On the other hand, MySQL has a more limited set of data types and is geared toward more straightforward web applications.
2. Geospatial Support: PostgreSQL strongly supports geospatial data, including a rich set of data types, functions, and operators for handling geographic data. MySQL, while it has some geospatial support, could be more robust in this area.
3. Indexing: In MySQL, the default index type is B-tree, which is well-suited for most use cases. PostgreSQL has a more refined indexing system than MySQL, including support for B-tree, GiST (Generalized Search Tree), and GIN (Generalized Inverted Index) indexes. These provide more options for optimizing query performance and data retrieval.
4. Replication: PostgreSQL and MySQL support database replication, but the methods and options for replication differ. PostgreSQL supports multi-master replication, while MySQL primarily supports master-slave replication. MySQL has recently introduced a new replication model called Group Replication, but it is still a relatively new feature with some limitations.
5. Transactions: PostgreSQL and MySQL InnoDB utilize MVCC (Multi-Version Concurrency Control) to handle concurrent data access. However, PostgreSQL offers evolved transaction management features such as transaction isolation levels, atomic transactions, and savepoints. In contrast, MySQL's transaction management options are more limited. PostgreSQL may be better for applications requiring high concurrency or complex transaction logic.
6. Stored Procedures: PostgreSQL and MySQL support stored procedures, but the language and functionality of stored procedures differ. PostgreSQL supports stored procedures written in various languages, including PL/pgSQL, PL/Tcl, PL/Perl, and more. MySQL, alternatively, primarily supports stored procedures written in SQL.
7. Extensions: PostgreSQL has a robust extension framework that allows developers to add custom functionality and extend the core features of the database. While it has some support for extensions, MySQL has a different level of extensibility than PostgreSQL.

Change data capture

In terms of change data capture (CDC), both MySQL binary logs and PostgreSQL write-ahead logs (WALs) can capture changes made to the database. However, the specific features and usage of the CDC may vary.

DBConvert Streams is a software that can read MySQL and PostgreSQL transaction logs and transform the records to another dialect, making it suitable for heterogeneous database replication in real time.

Performance

MySQL is known for its high performance and ability to process large amounts of data. It has been optimized for read-heavy workloads and has a fast indexing system that helps improve query performance. However, concurrency issues such as lock contention can occur when combined with write operations, leading to performance degradation. This is due to the implementation of locks at the table level, preventing all actions while the write operation is in progress.

The InnoDB storage engine is used to solve the table-level locking issue. It is one of the most popular and widely used storage engines in the MySQL ecosystem. InnoDB supports row-level locking, improving concurrency for mixed workloads.

Additionally, the recent development of the high-performance storage engine, MyRocks, has further improved MySQL's ability to handle write-intensive workloads.

PostgreSQL is designed to be more versatile, handling both read-heavy and write-heavy workloads, but with slightly lower performance than MySQL, which is optimized for read-heavy workloads. However, PostgreSQL has improved its performance in recent versions, especially regarding complex queries and data processing. Additionally, PostgreSQL has a more advanced indexing system compared to MySQL, which can improve performance for complex queries. PostgreSQL also supports advanced data types, like arrays and JSONB, which can result in more efficient data storage and retrieval. Ultimately, the performance of both PostgreSQL and MySQL will depend on various factors, such as hardware, data size, and query complexity.

When choosing between the two, consider your application's specific requirements and conduct performance testing with your data and workloads to determine the best fit.

Scalability

Both MySQL and PostgreSQL can scale, but they have different strengths and weaknesses when it comes to scalability. MySQL is often preferred for its horizontal scalability, which means it can be scaled out by adding more nodes to the database cluster. It is ideal for web applications that need to handle a large number of concurrent connections.

On the other hand, PostgreSQL is known for its vertical scalability, which means it can handle large amounts of data and processing power by adding more resources like memory and CPU to a single node. It also supports horizontal scaling through technologies such as sharding, which allows large datasets to be split across multiple nodes. PostgreSQL is preferred for applications requiring complex queries and transactions and for data warehouse and business intelligence workloads.

In terms of scalability, consider the specific requirements of your application. If you need to handle a large number of concurrent connections and need horizontal scalability, MySQL may be a better choice. However, PostgreSQL may be better if you require complex transactions and queries.

Cost

In 2023, it is still true that PostgreSQL is fully open-source and community-driven, whereas MySQL has a more complex history concerning licensing. MySQL was initially developed as a commercial product by MySQL AB, with free and paid versions available. The acquisition of MySQL AB by Oracle in 2010 raised some concerns among developers about the future of its open-source status. However, several open-source forks of the original MySQL, including MariaDB and Percona, have helped mitigate these concerns.

When to Use MySQL?

While PostgreSQL has many cutting-edge features and is often considered a more advanced and complex database management system than MySQL, it does have its drawbacks.

Some of the common weaknesses of PostgreSQL include the following:

1. Despite its advanced features and capabilities, PostgreSQL has yet to reach the level of popularity and widespread use of MySQL. This has resulted in a smaller number of third-party tools and a lesser number of experienced developers or database administrators in the PostgreSQL ecosystem.
2. Due to its advanced features, PostgreSQL can be more complex to set up and manage than MySQL, making it more suitable for experienced database administrators and developers.
3. PostgreSQL can be slower in certain use cases than MySQL due to its more complex architecture and features.
4. PostgreSQL can require more resources than MySQL, particularly in terms of memory and CPU usage. While PostgreSQL is open-source, the cost of implementation and maintenance can still be high due to its advanced features and increased resource requirements.
5. PostgreSQL forks a new process for each new client connection, which can allocate a significant amount of memory, typically around 10 MB per connection. However, this architecture is designed to provide improved security and isolation between different clients and is generally considered a trade-off for better performance, reliability, and scalability.
6. PostgreSQL is designed to prioritize extensibility, standards compliance, scalability, and data integrity. Sometimes, these features can decrease performance compared to MySQL, especially in simple read-heavy workloads. However, it's important to note that the exact performance difference depends on various factors, such as the data's size, the queries' complexity, and the hardware being used.

Which migration is more common: MySQL to PostgreSQL or PostgreSQL to MySQL?

The migration frequency between MySQL and PostgreSQL varies and depends on individual organizations' needs and requirements. Some organizations may migrate from MySQL to PostgreSQL to take advantage of its advanced features, better SQL compliance, and open-source compatibility. PostgreSQL is also more prevalent in specific industries, such as financial services, government, and data warehousing, where performance, scalability, and security are essential considerations.

On the other hand, other companies may migrate from PostgreSQL to MySQL for its simplicity, broad community of support, and lower cost of implementation.

The migration trend is towards moving from MySQL to PostgreSQL based on various indicators. These indicators suggest that more people are shifting from MySQL to PostgreSQL compared to the reverse.

1. Availability of migration tools: Many open-source and commercial migration tools are available to assist in migrating data from MySQL to PostgreSQL. At the same time, there are fewer tools to migrate data from PostgreSQL to MySQL.
2. Online resources: There is greater availability of online tutorials and resources on migrating from MySQL to PostgreSQL compared to the other way around.
3. Community growth: The PostgreSQL community has been growing faster than the MySQL community, indicating a growing interest in using PostgreSQL over MySQL.
4. Open source contributions: The number of open source contributions to PostgreSQL is increasing, indicating that more people are investing time and resources into the technology and finding it useful for their needs.
5. Enterprise adoption: Some of the world's largest and most data-intensive organizations, such as Cisco, Fujitsu, and the U.S. Federal Aviation Administration (FAA), have publicly stated that they have moved from MySQL to PostgreSQL.
6. User surveys: Industry analysts and database experts have conducted surveys that suggest that more people are considering or planning to switch from MySQL to PostgreSQL.

These facts only indicate that more migrations happen from MySQL to PostgreSQL than the other way around, and it may only be true in some cases.

Conclusion.

PostgreSQL and MySQL are robust relational database management systems with unique features and limitations. The decision to use one over the other should be based on the specific requirements of a project, such as the nature and amount of data, the intricacy of queries, and performance and scalability needs. As both PostgreSQL and MySQL are set to undergo further advancements in 2023, it is crucial to stay updated on their recent developments.

In addition, it's worth mentioning that tools like DBConvert Studio can help with migrating data between MySQL and PostgreSQL in either direction. These tools can simplify the process of transferring data from one database to another, which can be especially useful if you consider switching from one system to another.

title: Streaming data from MySQL to Postgres. published: true date: 2023-01-14 16:41:42 UTC tags: database,programming,mysql,postgresql

canonical_url: https://dbconvert.com/blog/streaming-data-mysql-postgres/

Streaming data from a MySQL database to a PostgreSQL database can be a useful way to move data between systems or to create a real-time replica of a database for reporting and analysis. One way to accomplish this is through the use of Change Data Capture (CDC) tools.

CDC is a method of tracking changes made to a database and capturing them in a separate stream. This stream can then be used to replicate the changes to another database.

DBConvert Streams helps to replicate your MySQL data to PostgreSQL in real-time. It captures data changes from a source MySQL database and applies them to a target PostgreSQL database. This can be done by setting up a source to read the binary log of a MySQL database and transform the changes to a format that can be consumed by target PostgreSQL database.

This blog post will provide a comprehensive guide on how to stream data from MySQL to Postgres.

To begin, let's clone the GitHub repository containing the MySQL to PostgreSQL streaming example.

git clone git@github.com:slotix/dbconvert-streams-public.git && cd dbconvert-streams-public/examples/mysql2postgres/sales-db/

Docker Compose configuration.

Since DBConvert Streams relies on multiple services, the most efficient way to start the containers is by using Docker Compose.

The docker-compose.yml file from the repository is provided below.

version: '3.9'
services:
  dbs-api:
    container_name: api
    image: slotix/dbs-api
    entrypoint:
      - ./dbs-api
      - --nats=nats:4222
      - --source=source-reader:8021
      - --target=target-writer:8022
    ports:
      - 8020:8020
    depends_on:
      - nats
    volumes:
      - ./mysql2pg.json:/mysql2pg.json:ro

  dbs-source-reader:
    container_name: source-reader
    image: slotix/dbs-source-reader
    entrypoint:
      - ./dbs-source-reader
      - --nats=nats:4222
    ports:
      - 8021:8021
    depends_on:
      - dbs-api

  dbs-target-writer:
    container_name: target-writer
    image: slotix/dbs-target-writer
    entrypoint:
      - ./dbs-target-writer
      - --nats=nats:4222
      - --prometheus=http://prometheus:9090
    ports:
      - 8022:8022
    depends_on:
      - dbs-source-reader

  nats:
    container_name: nats
    image: nats
    entrypoint: /nats-server
    command: "--jetstream -m 8222 --store_dir /data/nats-server"
    ports:
      - 4222:4222
      - 8222:8222

  prometheus:
    image: slotix/dbs-prometheus:latest
    container_name: prom
    user: root
    ports:
      - 9090:9090

  mysql-source:
    container_name: mysql-source
    build: ./source 
    environment:
      - MYSQL_ROOT_PASSWORD=123456
    ports:
      - '3306:3306'

  postgres-target:
    container_name: postgres-target
    image: postgres:15-alpine
    environment:
      - POSTGRES_USER=postgres
      - POSTGRES_PASSWORD=postgres
    command: postgres
    ports:
      - '5432:5432'

Docker-compose file for DBConvert Streams

This docker-compose file is going to bring up multiple services and link them together so they can communicate with each other. With this setup, the replication process takes place between mysql-source and postgres-target databases, and it is controlled by dbs-api, dbs-source-reader, dbs-target-writer and nats services.

DBConvert Streams services.

• dbs-api service is the entry point of DBConvert Streams. It is where requests are sent with configuration settings for the source and target databases. It specifies the connection details to other components of the system such as source-reader, and target-writer.
• dbs-source-reader service is responsible for monitoring and capturing changes in the source database, then sending batches of records to the Event Hub.
• dbs-target-writer service is used to receive changes from the Event Hub and apply them to the target database..
• nats service is the core of the Event Hub, it provides communication between other DBS services.
• prometheus service is used for monitoring the metrics of DBS services.

Database services.

The structure of the tables that will be used in our example is depicted in the diagram below.

Entity Relationship (ER) Diagram of Database.

The structure of MySQL source tables is adapted from the jdaarevalo/docker_postgres_with_data GitHub repository.

Source and Target Databases.

mysql-source database image is based on slotix/dbs-mysql:8, which includes all the necessary settings to enable MySQL CDC replication. This image also contains the initdb.sql script, which creates tables with the structures shown above.

postgres-target database, on the other hand, is based on the official lightweight postgres:15-alpine image. postgres-target database will receive all changes made to the mysql-source database.

Both of these databases, mysql-source and postgres-target, are typically located on separate physical servers in a production environment. However, in this example, we will run them on the same machine within distinct containers for demonstration purposes.

Execution.

To start all services described above, execute the following command:

docker-compose up --build -d

This command will use the docker-compose.yml file to build and start the necessary containers in detached mode (-d option). The --build flag will force the rebuild of the images before starting the containers.

To check if the MySQL database inside the running container has all tables created successfully by the script on start, you can run the command:

docker exec -it mysql-source mysql -uroot -p123456 source -e 'SHOW TABLES;'

By running the above command, you can see a list of tables created in the source database inside the container, which will confirm if the script has created all the tables successfully on start.

+------------------+
| Tables_in_source |
+------------------+
| city |
| country |
| order_status |
| product |
| sale |
| status_name |
| store |
| users |
+------------------+

List of tables created on source

Stream configuration

This is the stream configuration file mysql2pg.json which is used to setup the database replication process.

{
  "source": {
    "type": "mysql",
    "connection": "root:123456@tcp(mysql-source:3306)/source",
    "filter": {
      "tables": [
        { "name": "product", "operations": ["insert"]},
        { "name": "country", "operations": ["insert"]},
        { "name": "city", "operations": ["insert"]},
        { "name": "store", "operations": ["insert"]},
        { "name": "users", "operations": ["insert"]},
        { "name": "status_name", "operations": ["insert"]},
        { "name": "sale", "operations": ["insert"]},
        { "name": "order_status", "operations": ["insert"]}
      ]
    }
  },
  "target": {
    "type": "postgresql",
    "connection": "postgres://postgres:postgres@postgres-target:5432/postgres"
  },
  "limits": {
    "numberOfEvents": 0,
    "elapsedTime": 0
  }
}

• The source field specifies the type of the source database as "mysql" and the connection details to connect to the database, including username and password as well as the host and port.
• The filter field within the source field specifies that only certain tables and their corresponding operations (in this case "insert") will be replicated.
• The target field specifies the type of the target database as "postgresql" and the connection details to connect to the target database, including username and password as well as the host and port.
• The limits field specifies that number of events and elapsed time are set to zero (0), which means that there are no limits in place for this replication process.

It is also worth noting that according to this config, only insert operations will be captured on the tables specified in the filter field.

Send configuration to DBConvert Streams API.

docker run -t --rm \
    --network sales-db_default \
    curlimages/curl \
    --request POST \
    --url http://dbs-api:8020/api/v1/streams\?file=./mysql2pg.json

This command runs a docker run command to start a new container from the curlimages/curl image. It specifies that the container should join the network named sales-db_default using --network option.

This container will then run the command curl to make a HTTP POST request to the url http://dbs-api:8020/api/v1/streams?file=./mysql2pg.json. The URL contains an endpoint that is the dbs-api service which is running on port 8020 and is expecting a json file as a query parameter. This command creates a new stream on the dbs-api service with the configuration specified in the mysql2pg.json file.

It's important to note that this command assumes that the sales-db_default network and the dbs-api service are already created and running. It also assumes that the mysql2pg.json file is in the current working directory from which the command is run.

This is a JSON response indicating that the stream creation was successful.

{"status":"success",
    "data":{
        "id":"2KGlt8BCHLT0lXklrs5wqM6n7BQ",
        "source":{...},
        "target":{...},
        "limits":{}
     }
}

It contains the following fields:

• status: This field indicates the status of the request, in this case "success"
• data: This field contains the details of the stream that was created.
• id: This field contains a unique identifier for the stream, in this case "2KGlt8BCHLT0lXklrs5wqM6n7BQ"
• source: This field contains the details of the source database, including the type, connection details, and filter settings.
• target: This field contains the details of the target database, including the type and connection details.
• limits: This field contains the limits for the replication process, such as number of events and elapsed time.

Note that the details of the source and target field are not given here for brevity, it is just shown as ....

Check if tables on the target are created successfully.

💡 DBConvert Streams creates tables with the same structure as the source on the target if they are missing. At this point, all tables specified in the filter should exist on the Postgres target database.

To connect to the postgres-target Docker container and check if tables exist, you can run the following command:

 docker exec -it postgres-target psql -U postgres -d postgres -c '\dt'

By running the above command, you can see a list of tables created in the postgres-target database, which will confirm if DBConvert Streams has created all the tables successfully.

            List of relations
 Schema | Name | Type | Owner
--------+--------------+-------+----------
 public | city         | table | postgres
 public | country      | table | postgres
 public | order_status | table | postgres
 public | product      | table | postgres
 public | sale         | table | postgres
 public | status_name  | table | postgres
 public | store        | table | postgres
 public | users        | table | postgres
(8 rows)

Populate the source with sample data.

Now that the mysql-source and postgres-target databases have identical table sets with the same structure, it is time to find out if the streaming of data works properly. This can be done by inserting data into the mysql-source database and observing if the same data is replicated to the postgres-target database.

-- Set params
SET @number_of_sales = '100';
SET @number_of_users = '100';
SET @number_of_products = '100';
SET @number_of_stores = '100';
SET @number_of_countries = '100';
SET @number_of_cities = '30';
SET @status_names = '5';
SET @start_date = '2023-01-01 00:00:00';
SET @end_date = '2023-02-01 00:00:00';

USE source;

TRUNCATE TABLE city ;
TRUNCATE TABLE product ;
TRUNCATE TABLE country ;
TRUNCATE TABLE status_name;
TRUNCATE TABLE users;
TRUNCATE TABLE order_status;
TRUNCATE TABLE sale;
TRUNCATE TABLE store;

-- Filling of products
INSERT INTO product
WITH RECURSIVE t(id) AS (
    SELECT 1
    UNION ALL
    SELECT id + 1
    FROM t
    WHERE id + 1 <= @number_of_products
)
SELECT id, CONCAT_WS(' ','Product', id)
FROM t;

-- Filling of countries
INSERT INTO country
WITH RECURSIVE t(id) AS (
    SELECT 1
    UNION ALL
    SELECT id + 1
    FROM t
    WHERE id + 1 <= @number_of_countries
)
SELECT id, CONCAT('Country ', id)
FROM t;

-- Filling of cities
INSERT INTO city
WITH RECURSIVE t(id) AS (
    SELECT 1
    UNION ALL
    SELECT id + 1
    FROM t
    WHERE id + 1 <= @number_of_cities
)
SELECT id
    , CONCAT('City ', id)
    , FLOOR(RAND() * (@number_of_countries + 1))
FROM t;

-- Filling of stores
INSERT INTO store
WITH RECURSIVE t(id) AS (
    SELECT 1
    UNION ALL
    SELECT id + 1
    FROM t
    WHERE id + 1 <= @number_of_stores
)
SELECT id
    , CONCAT('Store ', id)
    , FLOOR(RAND() * (@number_of_cities + 1))
FROM t;

-- Filling of users
INSERT INTO users
WITH RECURSIVE t(id) AS (
    SELECT 1
    UNION ALL
    SELECT id + 1
    FROM t
    WHERE id + 1 <= @number_of_users
)
SELECT id
    , CONCAT('User ', id)
FROM t;

-- Filling of status_names
INSERT INTO status_name
WITH RECURSIVE t(status_name_id) AS (
    SELECT 1
    UNION ALL
    SELECT status_name_id + 1
    FROM t
    WHERE status_name_id + 1 <= @status_names
)
SELECT status_name_id
    , CONCAT('Status Name ', status_name_id)
FROM t;

-- Filling of sales  
INSERT INTO sale
WITH RECURSIVE t(sale_id) AS (
    SELECT 1
    UNION ALL
    SELECT sale_id + 1
    FROM t
    WHERE sale_id + 1 <= @number_of_sales
)
SELECT UUID() AS sale_id
    , ROUND(RAND() * 10, 3) AS amount
    , DATE_ADD(@start_date, INTERVAL RAND() * 5 DAY) AS date_sale
    , FLOOR(RAND() * (@number_of_products + 1)) AS product_id
    , FLOOR(RAND() * (@number_of_users + 1)) AS user_id
    , FLOOR(RAND() * (@number_of_stores + 1)) AS store_id
FROM t;

-- Filling of order_status
INSERT INTO order_status
WITH RECURSIVE t(order_status_id) AS (
    SELECT 1
    UNION ALL
    SELECT order_status_id + 1
    FROM t
    WHERE order_status_id + 1 <= @number_of_sales
)
SELECT UUID() AS order_status_id
    , DATE_ADD(@start_date, INTERVAL RAND() * 5 DAY) AS update_at
    , FLOOR(RAND() * (@number_of_sales + 1)) AS sale_id
    , FLOOR(RAND() * (@status_names + 1)) AS status_name_id
FROM t;

_filltables.sql script

This script above starts by truncating all the tables in the source database to clear any previous data, and then it inserts sample data into the tables. Each table is filled with sample data using a different set of parameters set at the top of the script.

To execute this SQL script, you can run the following command:

docker exec -i \
    mysql-source \
    mysql -uroot -p123456 -D source < $PWD/fill_tables.sql

Comparing number of records in source and target databases.

Let's compare the number of rows in the tables of the source and target databases.

docker exec -it mysql-source mysql -uroot -p123456 -D source -e "SELECT (SELECT COUNT(*) FROM product) as 'product_count',(SELECT COUNT(*) FROM country) as 'country_count',(SELECT COUNT(*) FROM city) as 'city_count',(SELECT COUNT(*) FROM store) as 'store_count',(SELECT COUNT(*) FROM users) as 'users_count',(SELECT COUNT(*) FROM status_name) as 'status_name_count',(SELECT COUNT(*) FROM sale) as 'sale_count',(SELECT COUNT(*) FROM order_status) as 'order_status_count';"

How many records are in each table of mysql-source db?

COUNTs for mysql-source tables.

docker exec -it postgres-target psql -U postgres -d postgres -c "SELECT (SELECT COUNT(*) FROM product) as product_count,(SELECT COUNT(*) FROM country) as country_count,(SELECT COUNT(*) FROM city) as city_count,(SELECT COUNT(*) FROM store) as store_count,(SELECT COUNT(*) FROM users) as users_count,(SELECT COUNT(*) FROM status_name) as status_name_count,(SELECT COUNT(*) FROM sale) as sale_count,(SELECT COUNT(*) FROM order_status) as order_status_count;"

how many records are in each table of postgres-target db?

COUNTs for postgres-target tables.

As you can see from the resulting output, all tables in both the source and target databases have an identical number of records in each table.

Check statistics.

The following command sends a GET request to the DBConvert API endpoint /api/v1/streams/stat which retrieves the statistics of current data stream. The jq command is used to format the JSON output for better readability.

docker run -t --rm \
    --network sales-db_default \
    curlimages/curl \
    --request GET \
    --url http://dbs-api:8020/api/v1/streams/stat | jq

{
  "streamID": "2KHTjpsZAUCb8y1BZny3YKoX5qO",
  "source": {
    "counter": 635,
    "elapsed": "0s",
    "started": "2023-01-13T17:24:50.089257721Z",
    "status": "RUNNING"
  },
  "target": {
    "counter": 635,
    "elapsed": "0s",
    "started": "2023-01-13T17:24:50.089649312Z",
    "status": "RUNNING"
  }
}

The above output shows the statistics of the current data stream. The "source" field shows the statistics of the source database, and "target" field shows the statistics of the target database.

The "counter" field shows the number of events that have been processed by the stream, "elapsed" field shows the time elapsed since the stream started, "started" field shows the date and time when the stream was started and "status" field shows the status of the current stream.

Prometheus metrics.

Prometheus is a monitoring system that scrapes metrics data from various sources and stores them in a time-series database. DBConvert Streams collects its internal metrics in Prometheus format, allowing you to explore and visualize live data in dashboards. To view the collected metrics, visit http://127.0.0.1:9090 in a web browser to access the Prometheus UI.

Conclusion

Change Data Capture (CDC) systems like DBConvert Streams can be used to stream data from a MySQL database to a PostgreSQL database in real-time, allowing you to keep the two systems in sync and take advantage of the unique features and capabilities of each database.

This guide provided information on streaming data in one direction, from MySQL Binlog to PostgreSQL. The DBConvert Streams Github repository contains more examples of configuring data streams, including from PostgreSQL Wals to MySQL and other configurations. These examples can serve as a starting point for setting up your own data stream, tailored to your specific needs and use case.

Getting your feedback about DBConvert Streams is essential for the development team to improve the software and make it more useful for the community. By sharing your ideas, reporting bugs, and requesting new features, you can actively participate in the development of the software and help to make it more robust and useful for everyone. Your participation is valued and appreciated by the development team and the community.

title: PostgreSQL Change data capture (CDC) + golang sample code. published: true date: 2022-03-28 09:12:40 UTC tags: database,postgresql,postgres,programming

canonical_url: https://dbconvert.com/blog/postgresql-change-data-capture-cdc/

The architecture of modern web applications consists of several software components such as dashboards, analytics, databases, data lakes, caches, search, etc.

The database is usually the core part of any application. Real-time data updates keep disparate data systems in continuous sync and respond quickly to new information. So how to keep your application ecosystem in sync? How do these other components get information about changes in the database? Change Data Capture or CDC refers to any solution that identifies new or changed data.

This post is about PostgreSQL CDC and the ways to achieve this.

Change data capture (CDC) is an approach to data integration to detect, capture, and deliver the changes made to database data sources.

In General, CDC-based data integration consists of the following steps:

1. Capture change data in a source database.
2. Convert the changed data to a format that your consumers can accept.
3. Publish the data to consumers or target database.

PostgreSQL offers two built-in ways to make CDC possible:

• From transaction logs, PostgreSQL WALs, aka Write Ahead Logs.
• Database Triggers.

Let's briefly discuss the pros and cons of using Transaction logs (WALs) and Triggers to capture data changes.

Triggers.

Trigger-based methods involve creating audit triggers on the database to capture all the events related to INSERT, UPDATE and DELETE methods.

Triggers can be attached to tables (partitioned or not) or views.

Triggers can also fire for TRUNCATE statements. If a trigger event occurs, the trigger's function is called at the appropriate time to handle the event.

• 😄 The most important advantage of this method is that all of this can be done at the SQL level, unlike transaction logs.
• 😕 However, the use of triggers has a significant impact on the performance of the source database, because these triggers need to be run on the application database when changes are made to the data.

Transaction Logs

On the other hand, for modern DBMS, transaction logs (WAL for PostgreSQL) are commonly used for transaction logging , replication, and REDO.

In PostgreSQL, all transactions like INSERT, UPDATE, DELETE, and DDLs are written to the WAL before the client sends the transaction result.

• The advantage of this approach is that it does not affect the performance of the database in any way.
• It also requires no modification to DB tables or the application. There is no need to create additional tables in the source database.
• Log-based CDC is generally considered the superior approach to change data capture applicable to all possible scenarios, including systems with extremely high transaction volumes.

If you want row-by-row streaming of Postgres data changes as they happen, you'll need Logical Decoding or Postgres logical replication feature.

Using Postgres Logical Decoding.

Logical decoding is the official name of PostgreSQL's log-based CDC (Logical Replication).

Logical decoding uses the content of the PostgreSQL Write-Ahead Log to store all of the activities that occur in the database. Write Ahead Log is an internal log that describes database changes on a storage level.

1. The first step in using logical decoding is to set the following parameters in the Postgres configuration postgresql.conf.

wal_level = logical
max_replication_slots = 5
max_wal_senders = 10

• Setting wal_level to logical allows the WAL to record information needed for logical decoding.
• Ensure that your max_replication_slots value is equal to or higher than the number of PostgreSQL connectors that use WAL plus the number of other replication slots your database uses.
• Ensure that the max_wal_senders parameter, which specifies the maximum number of concurrent connections to the WAL, is at least twice the number of logical replication slots. For example, if your database uses 5 replication slots in total, the max_wal_senders value must be 10 or greater.

Restart your Postgres server to apply the changes.

2. The second step is to set up logical replication using the output plugin test_decoding

Create a logical replication slot for the database you want to sync by running the following command.

SELECT pg_create_logical_replication_slot('replication_slot', 'test_decoding');

Note: Each replication slot has a name, which can contain lower-case letters, numbers, and the underscore character.

To verify the slot is created successfully run the following command.

SELECT slot_name, plugin, slot_type, database, active, restart_lsn, confirmed_flush_lsn FROM pg_replication_slots;

3. In the next step create a publication for all your tables or only specific ones. If you specify tables, you add or remove tables from the publication later.

CREATE PUBLICATION pub FOR ALL TABLES;

or

CREATE PUBLICATION pub FOR TABLE table1, table2, table3;

Optionally you can choose which operations to include in the publication. For example, the following publication includes only INSERT and UPDATE operations for table1.

CREATE PUBLICATION insert_update_only_pub FOR TABLE table1 WITH (publish = 'INSERT, UPDATE');

4. Verify that your chosen tables are in the publication.

psql-stream=# SELECT * FROM pg_publication_tables WHERE pubname='pub';
Output
pubname | schemaname | tablename
---------+------------+-----------
pub | public | table1
pub | public | table2
pub | public | table3
(3 rows)

Since then, our publication pub tracks all tables' changes in the psql-stream database.

5. Let's create an abstract table t and fill it with some records.

create table t (id int, name text);
INSERT INTO t(id) SELECT g.id FROM generate_series(1, 10) as g(id);

As a result we have 10 records in the table t.

psql-stream=# SELECT count(*) FROM t;
count
-------
10
(1 row)

6. Finally, it is time to check if our Logical Replication works.

Run the following command in the PostgreSQL console to see Postgres WAL entries.

SELECT * FROM pg_logical_slot_get_changes('replication_slot', NULL, NULL);

As a result, you get something like:

    lsn | xid | data                          
-----------+------+--------------------------------------------------------
 0/19EA2C0 | 1045 | BEGIN 1045
 0/19EA2C0 | 1045 | table public.t: INSERT: id[integer]:1 name[text]:null
 0/19EA300 | 1045 | table public.t: INSERT: id[integer]:2 name[text]:null
 0/19EA340 | 1045 | table public.t: INSERT: id[integer]:3 name[text]:null
 0/19EA380 | 1045 | table public.t: INSERT: id[integer]:4 name[text]:null
 0/19EA3C0 | 1045 | table public.t: INSERT: id[integer]:5 name[text]:null
 0/19EA400 | 1045 | table public.t: INSERT: id[integer]:6 name[text]:null
 0/19EA440 | 1045 | table public.t: INSERT: id[integer]:7 name[text]:null
 0/19EA480 | 1045 | table public.t: INSERT: id[integer]:8 name[text]:null
 0/19EA4C0 | 1045 | table public.t: INSERT: id[integer]:9 name[text]:null
 0/19EA500 | 1045 | table public.t: INSERT: id[integer]:10 name[text]:null
 0/19EA5B0 | 1045 | COMMIT 1045
(13 rows)

pg_logical_slot_peek_changes is another PostgreSQL command to peek changes from WAL entries without consuming them. So calling pg_logical_slot_peek_changes multiple times returns the same result each time.

On the other hand pg_logical_slot_get_changes only returns results the first time. The following calls of pg_logical_slot_get_changes return empty result sets. This means when get command is executed, the results are served and deleted, which greatly enhancing our ability to write the logic for using these events to create a replica of the table.

7. Remember to destroy a slot you no longer need to stop it consuming

SELECT pg_drop_replication_slot('replication_slot');

Output plugins.

We've already talked about test_decoding output plugin available on Postgres 9.4+. Though created as an example of an output plugin, it is still useful if your consumer supports it.

Along with the test_decoding plugin, another pgoutput plugin is shipped with PostgreSQL natively. pgoutput is available since Postgres 10. Some consumers support it for decoding (e.g. Debezium).

Run the following command to create the plugin based on pgoutput as in step 2 above.

SELECT * FROM pg_create_logical_replication_slot('replication_slot', 'pgoutput');

The following command consumes data changes similar to those described in step 6.

psql-stream=# SELECT * FROM pg_logical_slot_peek_binary_changes('replication_slot', null, null, 'proto_version', '1', 'publication_names', 'pub');
    lsn | xid | data                                           
-----------+------+------------------------------------------------------------------------------------------
 0/19A15F8 | 1038 | \x4200000000019a1d9000027de20a91a0ea0000040e
 0/19A15F8 | 1038 | \x52000080387075626c69630074006400020169640000000017ffffffff006e616d650000000019ffffffff
 0/19A15F8 | 1038 | \x49000080384e0002740000000234306e
 0/19A1890 | 1038 | \x49000080384e0002740000000234316e
 0/19A1910 | 1038 | \x49000080384e0002740000000234326e
 0/19A1990 | 1038 | \x49000080384e0002740000000234336e
 0/19A1A10 | 1038 | \x49000080384e0002740000000234346e
 0/19A1A90 | 1038 | \x49000080384e0002740000000234356e
 0/19A1B10 | 1038 | \x49000080384e0002740000000234366e
 0/19A1B90 | 1038 | \x49000080384e0002740000000234376e
 0/19A1C10 | 1038 | \x49000080384e0002740000000234386e
 0/19A1C90 | 1038 | \x49000080384e0002740000000234396e
 0/19A1DC0 | 1038 | \x430000000000019a1d9000000000019a1dc000027de20a91a0ea
(13 rows)

Here you can notice that the results are returned in binary format. pgoutput plugin produces binary output.

wal2json is another popular output plugin for logical decoding.

Here is a sample output from wal2json plugin

{
      "change":[
         {
            "kind":"insert",
            "schema":"public",
            "table":"t",
            "columnnames":[
               "id",
               "name"
            ],
            "columntypes":[
               "integer",
               "character varying(255)"
            ],
            "columnvalues":[
               1,
               ""
            ]
         }
      ]
   }
   {
      "change":[
         {
            "kind":"update",
            "schema":"public",
            "table":"t",
            "columnnames":[
               "id",
               "name"
            ],
            "columntypes":[
               "integer",
               "character varying(255)"
            ],
            "columnvalues":[
               1,
               "New Value"
            ],
            "oldkeys":{
               "keynames":[
                  "id"
               ],
               "keytypes":[
                  "integer"
               ],
               "keyvalues":[
                  1
               ]
            }
         }
      ]
   }
   {
      "change":[
         {
            "kind":"delete",
            "schema":"public",
            "table":"t",
            "oldkeys":{
               "keynames":[
                  "id"
               ],
               "keytypes":[
                  "integer"
               ],
               "keyvalues":[
                  1
               ]
            }
         }
      ]
   }

Important tips about slots.

Keep the following in mind when working with slots:

• Each slot has only one output plugin (you choose which one).
• Each slot provides changes from only one database.
• One database can have multiple slots.
• Each data change is typically is emitted once per slot.
• But a slot may re-emit the changes when the Postgres instance is restarted. A consumer must deal with this situation.
• An unconsumed slot is a threat to the availability of your Postgres instance. Postgres will save all WAL files for these unconsumed changes. This can lead to storage overflow.

PostgreSQL WAL consumers.

A consumer is any application that can ingest the Postgres logical decoding stream. pg_recvlogical is a PostgreSQL application that can manage slots and consume the stream from them. It is included in the Postgres distribution, so it's probably already installed with PostgreSQL.

Photo by Markus Spiske / Unsplash

Golang sample code.

The following Golang code example shows how to get started creating your own Postgress WAL consumer. It uses PostgreSQL-10.x logical replication to stream database changes (decoded WAL messages) from the source database.

package main

import (
    "context"
    "fmt"
    "os"
    "os/signal"
    "strings"
    "time"

    "github.com/jackc/pgconn"
    "github.com/jackc/pglogrepl"
    "github.com/jackc/pgproto3/v2"
)

const CONN = "postgres://postgres:postgres@localhost/psql-streamer?replication=database"
const SLOT_NAME = "replication_slot"
const OUTPUT_PLUGIN = "pgoutput"
const INSERT_TEMPLATE = "create table t (id int, name text);"

var Event = struct {
    Relation string
    Columns []string
}{}

func main() {
    ctx, cancel := signal.NotifyContext(context.Background(), os.Interrupt)
    defer cancel()
    conn, err := pgconn.Connect(ctx, CONN)
    if err != nil {
        panic(err)
    }
    defer conn.Close(ctx)

    // 1. Create table
    if _, err := conn.Exec(ctx, INSERT_TEMPLATE).ReadAll(); err != nil {
        fmt.Errorf("failed to create table: %v", err)
    }

    // 2. ensure publication exists
    if _, err := conn.Exec(ctx, "DROP PUBLICATION IF EXISTS pub;").ReadAll(); err != nil {
        fmt.Errorf("failed to drop publication: %v", err)
    }

    if _, err := conn.Exec(ctx, "CREATE PUBLICATION pub FOR ALL TABLES;").ReadAll(); err != nil {
        fmt.Errorf("failed to create publication: %v", err)
    }

    // 3. create temproary replication slot server
    if _, err = pglogrepl.CreateReplicationSlot(ctx, conn, SLOT_NAME, OUTPUT_PLUGIN, pglogrepl.CreateReplicationSlotOptions{Temporary: true}); err != nil {
        fmt.Errorf("failed to create a replication slot: %v", err)
    }

    var msgPointer pglogrepl.LSN
    pluginArguments := []string{"proto_version '1'", "publication_names 'pub'"}

    // 4. establish connection
    err = pglogrepl.StartReplication(ctx, conn, SLOT_NAME, msgPointer, pglogrepl.StartReplicationOptions{PluginArgs: pluginArguments})
    if err != nil {
        fmt.Errorf("failed to establish start replication: %v", err)
    }

    var pingTime time.Time
    for ctx.Err() != context.Canceled {
        if time.Now().After(pingTime) {
            if err = pglogrepl.SendStandbyStatusUpdate(ctx, conn, pglogrepl.StandbyStatusUpdate{WALWritePosition: msgPointer}); err != nil {
                fmt.Errorf("failed to send standby update: %v", err)
            }
            pingTime = time.Now().Add(10 * time.Second)
            //fmt.Println("client: please standby")
        }

        ctx, cancel := context.WithTimeout(ctx, time.Second*10)
        defer cancel()

        msg, err := conn.ReceiveMessage(ctx)
        if pgconn.Timeout(err) {
            continue
        }
        if err != nil {
            fmt.Errorf("something went wrong while listening for message: %v", err)
        }

        switch msg := msg.(type) {
        case *pgproto3.CopyData:
            switch msg.Data[0] {
            case pglogrepl.PrimaryKeepaliveMessageByteID:
            //    fmt.Println("server: confirmed standby")

            case pglogrepl.XLogDataByteID:
                walLog, err := pglogrepl.ParseXLogData(msg.Data[1:])
                if err != nil {
                    fmt.Errorf("failed to parse logical WAL log: %v", err)
                }

                var msg pglogrepl.Message
                if msg, err = pglogrepl.Parse(walLog.WALData); err != nil {
                    fmt.Errorf("failed to parse logical replication message: %v", err)
                }
                switch m := msg.(type) {
                case *pglogrepl.RelationMessage:
                    Event.Columns = []string{}
                    for _, col := range m.Columns {
                        Event.Columns = append(Event.Columns, col.Name)
                    }
                    Event.Relation = m.RelationName
                case *pglogrepl.InsertMessage:
                    var sb strings.Builder
                    sb.WriteString(fmt.Sprintf("INSERT %s(", Event.Relation))
                    for i := 0; i < len(Event.Columns); i++ {
                        sb.WriteString(fmt.Sprintf("%s: %s ", Event.Columns[i], string(m.Tuple.Columns[i].Data)))
                    }
                    sb.WriteString(")")
                    fmt.Println(sb.String())
                case *pglogrepl.UpdateMessage:
                    var sb strings.Builder
                    sb.WriteString(fmt.Sprintf("UPDATE %s(", Event.Relation))
                    for i := 0; i < len(Event.Columns); i++ {
                        sb.WriteString(fmt.Sprintf("%s: %s ", Event.Columns[i], string(m.NewTuple.Columns[i].Data)))
                    }
                    sb.WriteString(")")
                    fmt.Println(sb.String())
                case *pglogrepl.DeleteMessage:
                    var sb strings.Builder
                    sb.WriteString(fmt.Sprintf("DELETE %s(", Event.Relation))
                    for i := 0; i < len(Event.Columns); i++ {
                        sb.WriteString(fmt.Sprintf("%s: %s ", Event.Columns[i], string(m.OldTuple.Columns[i].Data)))
                    }
                    sb.WriteString(")")
                    fmt.Println(sb.String())
                case *pglogrepl.TruncateMessage:
                    fmt.Println("ALL GONE (TRUNCATE)")
                }
            }
        default:
            fmt.Printf("received unexpected message: %T", msg)
        }
    }
}

This code just logs incoming events, but in a production environment you can easily send them to a message queue or a target database.

Conclusion

Logical decoding in PostgreSQL provides an efficient way for other application components to stay up-to-date with data changes in your Postgres database.

Traditionally, the pull notification model has been used, in which each application component queries Postgres at a certain interval. Logical encoding uses the push notification model , where Postgres notifies other parts of the application of every change as soon as it happens.

Data change events can now be sent to consumers in milliseconds without querying the database. With logic decoding, the PostgreSQL database becomes a central part of your modern dynamic real-time application.

DBConvert Streams is a new data replication platform that uses CDC technology to stream data between databases.

title: How to scrape a website that requires login? published: true date: 2020-11-09 22:04:24 UTC tags: cookies,webscraping,webdev,showdev

canonical_url: https://dataflowkit.com/blog/how-to-pass-cookies-to-a-scraper-from-a-web-browser/

Many websites demand users to register or login before they provide any information. Browsers store cookies for each session as the user navigates the website. Other websites may show pop-ups if they do not have location cookies or if the user explicitly consents to the collection of their data.

Indeed, you can simulate user input to enter credentials, click a button to submit a form. Sometimes, you need to press a checkbox to accept website terms while scraping the data.

Another way is to pass session cookies when you send a request to a website. This article will show how to transfer cookies from a web browser to a Dataflow Kit web scraper.

Follow the instructions described below to crawl specific websites that require login:

1. Install EditThisCookie extension to your web browser.
2. Go to the website that you want to crawl and sign in with your credentials.

3. Open the "EditThisCookie" extension by clicking the button next to your URL. Copy the cookies to the clipboard using the "Export" button.

4. Now paste cookies (Ctrl + V) from the clipboard into the "Initial cookies" field of a Dataflow Kit scraper. Cookies in JSON array format are compatible with the cookie format used by Dataflow Kit.

As an example, we'll use the Dataflow Kit Screen Capture Service to illustrate the cookie transfer function.

That's all! Now you run the scraper, and it starts already logged in.

Result

In the captured screenshot, we can see that it was captured after the login page.

You are not limited to transferring the initial cookies only to the Dataflow Kit services provided on our website. You can add initial cookies to any custom web scraper powered by the Dataflow Kit framework. You can customize your payloads at https://account.dataflowkit.com/tasks

You can customize any Task payload and add InitialCookies manually to request.

Depending on a scraped website, cookies may be short-lived, and this approach with passing initial cookies is not a way to go. In this case, the right solution is to use actions to simulate filling out forms and pressing the submit button.

title: Beginner’s guide to Web Scraping with PHP. published: true date: 2020-11-02 15:25:48 UTC tags: PHP,API,scrape,webscraping

canonical_url: https://dataflowkit.com/blog/php-api/

Scraping simple HTML from the Web is not a problem in modern programming languages. While PHP is especially suited for web development, its ability to send HTTP requests is severely lacking. The Requests library for PHP developers is an excellent solution for sending HTTP requests to websites.

Render Javascript webpages.

It's not enough to download content from a website as-is with a library like PHP Requests. Most modern websites are heavily based on Javascript frameworks like Angular, React, or Vue.js. We can talk about client-side rendering where Javascript content returned as a response needs to be rendered in a web browser. Mainly the headless Chrome browser is used to render dynamic content from websites and returns it as a static HTML.

Using Proxies.

Another challenge is to fetch web page content restricted to users from specified countries only. Using proxy servers is required to obtain country-specific versions of target websites or to bypass content download restrictions.

In most cases, running your own headless Chrome browser cluster and a proxy pool is expensive. It makes more sense to use a special service to
render Javascript driven web pages in the cloud, and return static HTML.

How to scrape Websites using PHP script?

I will share the code for PHP developers who have shown a lot of interest lately. We will generate the simple PHP script that sends requests to Web Scraping Service API. To automate HTML scraping tasks, follow the steps describes below:

1. Get a Free API Key.

Dataflow Kit API Key is required to get access to Dataflow Kit API. the server. You can obtain it from the user dashboard after free registration. Once you sign-up, we grant you free 1000 credits.

Go to https://account.dataflowkit.com and either use Facebook/Google login or register with your email.

Click on the "Log in" button to register with your Facebook or Google account. Or press the "Sign Up" link to register with your email.

You will need to authorize requests to the Dataflow Kit API. Later we will add it to our PHP script. Please find it in dashboard Settings.

2. Install PHP Requests.

There is one dependency here. Before running the final script, follow installation instructions at https://github.com/rmccue/Requests and install PHP Requests package mentioned above.

3. Generate PHP script and send requests to the API.

3.1. Go to https://dataflowkit.com/html-scraping . Specify some parameters for HTML Scraping API code generator to generate PHP Script.

Depending on specified parameters, you get something like:

<?php
include('vendor/rmccue/requests/library/Requests.php');
Requests::register_autoloader();
$headers = array(
    'Content-Type' => 'application/json'
);
$url = 'https://dbconvert.com/';
$data = '{"url":"' . $url . '","type": "chrome","waitDelay": 0.5}';
$response = Requests::post('https://api.dataflowkit.com/v1/fetch?api_key=API-KEY', $headers, $data);
var_dump($response->body);
?>

3.2. Save the code above, for example, as "dfk-api.php"

3.3. Now add the actual API Key found at https://account.dataflowkit.com/settings in place of API-KEY. It looks like something "ab5cc2a84f7efab1693e8fc72he5f7e844b1bf5cbad9ea33". See the step #1.

3.4. That's all. Now you can run the script and get rendered HTML content from any web site.

Note for Docker users.

It is even simpler to build and run a docker image to run the script.

FROM php:7.4-cli
RUN apt-get update && \
    apt-get upgrade -y && \
    apt-get install -y git
RUN git clone git://github.com/rmccue/Requests.git
COPY . /usr/src/dfk-api
RUN mv Requests /usr/src/dfk-api
WORKDIR /usr/src/dfk-api
CMD ["php", "./dfk-api.php"]

Follow the steps below to build & run a PHP script that calls Dataflow Kit HTML scraping API service:

1. Open file dfk-api.php
2. Exchange API-KEY with the actual one from https://account.dataflowkit.com/settings . You can obtain it for free after registration at https://dataflowkit.com
3. Run the following command in the terminal to build a docker image.

   docker build -t dfk-api-php .
   

4. Run a command in a new container

   docker run -it --rm --name dfk-api-php dfk-api-php
   

Github repository for PHP code for accessing Dataflow Kit API.

Feel free to fork a Github repository at https://github.com/slotix/dfk-api-php and customize the code for your needs.

Conclusion

Web scraping of plain HTML web pages generated by a server is simple. You can use "PHP Requests" library to get HTML content.

When scraping large amounts from dynamically generated Javascript data, you might run into the following problems:

• You need to run multiple instances of the headless Chrome browser to handle large amounts of input.
• You have to send requests through a pool of proxies to avoid blocking.

In the same way, you can create PHP scripts to save web pages as PDF or take screenshots.

The next step obviously after scraping a webpage is to extract specific data from rendered HTML. Depending on a website, it may be a separate HTML element like an image, text, link. Or for example, e-commerce sites list several products on a page as blocks of data grouped by some patterns.

You can use other PHP code generators available on dedicated pages to build PHP scripts to make requests for scraping various web sites.

title: Scrape Google Maps: Extract business leads, phone numbers, addresses. published: true date: 2020-08-17 12:16:10 UTC tags: googlemaps,googlemapsscraping

canonical_url: https://dataflowkit.com/blog/scrape-google-maps-extract-business-leads-phone-numbers-addresses/

A lot of places on the Internet contain many lead generation sources that you can use to market your product or service. Google Maps is probably one of the best to get information about local businesses because of almost all companies presented in Google Maps.

It would take you a lot of time to manually copy and paste data from Google Maps. We offer a better way to scrape Google maps data to CSV, Excel, or JSON data.

If you're not a coder, it is enough to enter a search query and customize several parameters before launching Google Maps Scraper and getting desired data.

Or you can generate a "ready-to-run" code for your preferred language to send requests to Google scraper API.

Extract 100% Relevant B2B Leads Data Using Google Maps Extractor to Increase Your Sales Revenue.

Google maps scraping service from Dataflow Kit automates data extraction of the following fields:

1. Business Name
2. Address
3. Phone Number
4. Website
5. Rating
6. Reviews

Please check out Google maps scraper.