top of page

Blog


In today's rapidly evolving data landscape, the ability to efficiently replicate and manage vast datasets is crucial for organizations seeking to harness the power of cloud analytics. Informatica's Mass Ingestion Service, renowned for its robust data replication capabilities, has taken a significant leap forward with the introduction of SuperPipe technology for Snowflake.


This groundbreaking innovation promises to revolutionize the way Informatica users replicate data into Snowflake, boasting speeds up to 3.5 times faster, dramatically reduced costs, and significantly lower latency. In this demonstration, we'll explore how to configure Informatica's Mass Ingestion Service to seamlessly replicate data from SQL Server into Snowflake, leveraging the full potential of Superpipe. Whether you're a seasoned data professional or new to the cloud data platform, this post will equip you with the knowledge to make the most of this exciting new technology, setting the stage for enhanced data analytics capabilities.


What is SuperPipe?


SuperPipe is Informatica's technology for harnessing Snowflake's Snowpipe API, this feature has enabled several breakthroughs for users replicating large volumes of data into Snowflake:


  • Low latency ingestion

  • Ordered ingestion

  • High Scalability

  • Query results in near real-time

  • Cost reduction


Besides the performance enhancements, a great reason to start using SuperPipe with Informatica is it will end up reducing your operating costs. Snowpark API calls do not utilize your compute data warehouse and therefore does not contribute to running costs.


The team at Automate Vision have put together an overview of how to get started with this technology using your own Informatica IDMC instance. Enjoy the overview below and reach out to Automate Vision for your Informatica needs.



Thank you for taking the time to explore the solution concept and experience the demo. We hope this demonstration has provided valuable insights into how our product can address your needs effectively. If you have any questions or require further assistance, please don't hesitate to reach out to Automate Vision. Whether you're looking to optimize your Informatica needs or explore other solutions, we're here to help you succeed. Thank you once again for your time, and we look forward to collaborating with you in the near future.



The product that kicked off the ETL revolution in the 1990s is a workhorse in thousands of enterprises today, powering significant data warehouses across financial services, banking, government, retail, insurance, healthcare, and manufacturing essentially every industry has been touched at some point by PowerCenter.


With so much changing across the data management landscape, it's rare to see something so persistent. When Informatica Cloud (aka IICS aka IDMC) was released to the market it opened up enterprise-class data integration to new market segments that previously would not have considered PowerCenter or could afford it. Over time IDMC has become a powerhouse in data management offering everything an individual team or sophisticated IT organization could ask for.


What Has Changed in 2024?


New conversion options:


  • Informatica Modernization Team - Informatica can take on your migration project for you. They will have access to the latest tooling however this comes at a premium far above market rates.

  • Conversion With Specialist Tooling with Partner - Whether using Informatica's internal conversion tooling or Automate Vision's Converter. This remains the most flexible option for quickly converting a complex PowerCenter environment to IDMC. We utilize several tools to ensure a smooth migration and Automate Vision is significantly more affordable than alternatives which previously was an impediment to migration projects.

  • CDI for PC with Migration Workbench - This is the latest option from Informatica. It essentially 'cloudifies' your PowerCenter environment meaning you no longer need to deal with upgrades and converts your contract to IPU billing. This is a good option to start on your journey however the DIY approach with the Workbench still requires specialist knowledge to know how to use it. It is up to each customer to decide whether they want to become experts for a one-off migration or work with Automate Vision to do this and focus on becoming experts in IDMC.

  • Hybrid Approach - We believe the best option for the majority of customers is a hybrid approach of leveraging the Informatica Modernization team for high level planning but working with a partner to execute the conversion for you. This ensures early issues are caught and allows you to focus on the future with IDMC.


Our experience has been that most customers want to focus on their data and business. The challenge with a migration is after the project is completed the investment in those skills is no longer needed by the business. This is where we come in to provide experienced consultants who can get the migration completed the first time.


In 2024 What Benefits Exist To IDMC?



#1 You're Investing Into The Modern Data Stack (Cloud Data Warehouse, Cloud Ecosystem, Cloud Dashboards)

Probably the number 1 reason to consider migrating your PowerCenter instance to IDMC is if you're moving the center of gravity for your data infrastructure to the cloud. Spend some time reviewing the product updates pages for IDMC, Snowflake, Databricks, Redshift or any cloud application. You'll notice these platforms receive significant amounts of R&D investment and hence new features are released every 3 months. IDMC tracks these changes and keeps your investment up to date so you can get the best out of your data. The most recent example of this in action is Superpipe for Snowflake. Superpipe enables 3.5x faster data transfer speeds and is fully supported by IDMC. This feature is simply turned on and available without any manual efforts.


#2 Future Proof Your Transformation Logic

As mentioned above, there are now no impediments to converting your mappings with so many options available.


#3 A Platform For Data Management

Its debatable whether moving an ETL tool for an ETL tool in the cloud is worth the trouble. But if you're investing in the cloud you're investing for future growth which means you're eventually going to need things like data quality, realtime integration, data distribution and governance. Instead of attempting to cobble 8 different enterprise apps together and then get them to cooperate IDMC presents a single pane of glass where user access, UX, support, knowledge, collaboration are uniform across applications.


#4 Scalability and Elasticity

If you need more processing power in PowerCenter this means manualIy installing additional servers to scale the environment. IDMC can scale up or down based on demand, making it well-suited for handling variable workloads with nothing more than configuration. Customers can easily adapt to changing data integration requirements without the need to invest in additional on-premises infrastructure.


#5 Lower Total Cost of Ownership (TCO)

Moving to the cloud will reduce infrastructure maintenance, upgrade and support costs, as there is no need to purchase, maintain, and upgrade physical hardware and software. Customers can benefit from a subscription-based pricing model, which can be more cost-effective in the long run. The majority of work done in IDMC is clicking and navigating through web interfaces. The infrastructure maintenance cost goes to zero.


#6 Faster Deployment

IDMC offers rapid deployment, enabling customers to get started with data integration projects more quickly compared to traditional on-premises solutions. This agility can be especially important in fast-paced business environments. This compounds when adding new connectors (via the connector store) or simply using new services within IDMC. Because of the IPU consumption approach all services are made available to you.


#7 Global Accessibility

We live in a remote world and working from home during the pandemic accelerated that. IDMC is accessible from anywhere with an internet connection. This is advantageous for organizations with distributed teams or the need to access and integrate data from various geographic locations.


#8 Hybrid and Multi-Cloud Integration

IDMC supports hybrid integration scenarios, allowing customers to seamlessly connect and integrate data across on-premises and cloud-based applications, databases, and platforms. It also offers multi-cloud capabilities, enabling integration with data stored in different cloud providers.


#9 Security and Compliance

Many PowerCenter customers mistakenly assume data processed by IDMC goes through third-party data centers beyond their firewall. IDMC provides processing behind your firewall in addition to elastic processing and pushdown. All of this is wrapped in robust security features, including encryption, authentication, and access controls. It also helps customers adhere to compliance regulations by offering features like data masking and auditing.


#10 Data Quality and Governance

IDMC includes data quality and data governance capabilities, allowing customers to ensure the accuracy, consistency, and reliability of their data as it moves between systems and applications. There is no need to manually import assets between external products as everything is built in.


#11 Real-Time Integration

IDMC supports real-time data integration and streaming, enabling customers to make informed decisions based on up-to-the-minute data.


While this list is not exhaustive, we hope it gives you some food for thought on the options that exist and what has changed in 2024 for PowerCenter migrations.


Please reach out to us if you are considering a migration or want to explore the best options for your business.


  • Steve Mitchell, CEO

Welcome to the inaugural post of our blog. This is our space on the web to contribute our thoughts and learnings in an unfiltered way on everything related to data, analytics, technology and the industry. We want this to be a space that adds value to our industry and can become a reference point for how we and the industry evolve over time. We cant guarantee it will always stay on topic but it absolutely will be interesting.


Why "Automate Vision" ?


The name came from our shared experiences in automating the multitude of problems in the data industry along with the first product we ever built was a tool to automate conversion of metadata assets. Automation is a key tenant of the technologies we work with, taking the complex and mundane and turning them into small automated steps that are easily managed and executed without effort. This clearly describes what our customers have been trying to achieve and hence Automate Vision was born.


As Automate Vision we bring together skilled analysts, consultants, architects and data engineers with strong opinions on data management & analytics that want to see their work deliver success to customers. We feel that similar to us, our customers want to engage on bold ideas, be challenged and feel a shared interest in solving hard problems together. Our goal is to match our tenacity for solving data problems with customers taking on the biggest data challenges in their industry and automate those problems according to their vision.


Our platform of choice for delivering these solutions is Informatica's Intelligent Data Management Cloud (IDMC). We settled on IDMC after decades of our own experiences working with the platform and alternatives. Every year we've seen a proliferation in coding languages, data platforms, open source projects and proprietary products, with each iteration it gets harder for the individual organization to select the right platform, find the right team, maintain legacy code and keep the lights on. As much as we love coding "hello world!" in a freshly created technology it can get in the way of solving human problems as complexity grows and worse takes us away from enjoying being human. Anyone left debugging several hundred lines of SQL code over a weekend due to a commodity problem can attest to this.


Commodity problems are patterns that exist within every business. They arent unique to your business but exist across millions of business environments. They are either quickly resolved using tried and true techniques or become the swamp your wagons remain in.


Examples of commodity problems:


  • Ingesting data

  • Moving data between environments such as on premises to cloud

  • Catching null values in critical columns

  • Transforming data in place

  • Consuming data within dashboards

  • Exposing data via api

  • Documenting where data for that dashboard came from

  • Deduplicating records


This is why a platform such as IDMC is compelling to a business that wants to focus on its business. Each part of the platform relates to a key architectural aspect of data management and contributes to automating a commodity problem. Because of the platform approach as your data management vision grows the next building block is already there and ready to go.


Thats our vision, lets get started on yours.





bottom of page