{"id":540,"date":"2023-07-08T12:14:58","date_gmt":"2023-07-08T02:14:58","guid":{"rendered":"https:\/\/andyhowes.local\/?p=540"},"modified":"2023-12-28T04:41:46","modified_gmt":"2023-12-27T17:41:46","slug":"easily-connect-to-dataverse-microsoft-dynamics-with-python","status":"publish","type":"post","link":"https:\/\/andyhowes.local\/easily-connect-to-dataverse-microsoft-dynamics-with-python\/","title":{"rendered":"Easily connect to Dataverse\/Microsoft Dynamics with Python"},"content":{"rendered":"\n<p>Discover the power of connecting to Dataverse, Microsoft Dynamics and Power Apps with Python and unlock a wide range of productivity benefits. <\/p>\n\n\n\n<p>In this article, I show you how to use the program I developed to enable easy, authenticated access to your Dynamics environment&#8217;s OData API using Python.<\/p>\n\n\n\n<div class=\"wp-block-rank-math-toc-block\" id=\"rank-math-toc\"><h2>Contents<\/h2><nav><ul><li class=\"\"><a href=\"#why-connect-with-python\">Why connect with Python<\/a><\/li><li class=\"\"><a href=\"#connecting-to-dataverse-with-python\">Connecting to Dataverse with Python<\/a><\/li><li class=\"\"><a href=\"#key-uses\">Key uses<\/a><\/li><li class=\"\"><a href=\"#step-by-step-guide-to-connect-to-dataverse\">Step-by-step guide to connect to Dataverse<\/a><ul><li class=\"\"><a href=\"#step-1-create-a-new-azure-active-directory-app\">Step 1: Create a new Azure Active Directory app<\/a><\/li><li class=\"\"><a href=\"#step-2-grant-app-permissions-for-dynamics-crm\">Step 2: Grant app permissions for Dynamics CRM<\/a><\/li><li class=\"\"><a href=\"#step-3-configure-authentication\">Step 3: Configure authentication<\/a><\/li><li class=\"\"><a href=\"#step-4-get-app-information\">Step 4: Get app information<\/a><\/li><li class=\"\"><a href=\"#step-5-clone-the-repo\">Step 5: Clone the repo<\/a><\/li><li class=\"\"><a href=\"#step-6-configure-your-environment-json\">Step 6: Configure your environment JSON<\/a><\/li><li class=\"\"><a href=\"#step-7-use-your-favourite-ide-to-interact-with-dataverse-with-python\">Step 7: Use your favourite IDE to interact with Dataverse with Python\u00a0<\/a><\/li><\/ul><\/li><li class=\"\"><a href=\"#py-connect-dataverse\">PyConnectDataverse<\/a><\/li><\/ul><\/nav><\/div>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"why-connect-with-python\">Why connect with Python<\/h2>\n\n\n\n<p>There are lots of reasons to connect to your Dataverse data with Python.<\/p>\n\n\n\n<p>For example, when building a new Microsoft Dynamics 365 solution or using Dataverse with Power Apps, often organisational data needs to be imported into Dataverse. I&#8217;ve needed to do this often as part of my work <a href=\"https:\/\/andyhowes.local\/improving-outcomes-with-a-crm\/\" data-type=\"post\" data-id=\"329\">developing a CMS for Femili PNG<\/a>.<\/p>\n\n\n\n<p>Unfortunately, the available options like <a href=\"https:\/\/learn.microsoft.com\/en-us\/power-apps\/maker\/data-platform\/self-service-data-prep-with-dataflows\" data-type=\"URL\" data-id=\"https:\/\/learn.microsoft.com\/en-us\/power-apps\/maker\/data-platform\/self-service-data-prep-with-dataflows\" target=\"_blank\" rel=\"noreferrer noopener\">Power Query Dataflows<\/a> and the <a href=\"https:\/\/learn.microsoft.com\/en-us\/power-apps\/maker\/data-platform\/data-platform-excel-addin\" data-type=\"URL\" data-id=\"https:\/\/learn.microsoft.com\/en-us\/power-apps\/maker\/data-platform\/data-platform-excel-addin\" target=\"_blank\" rel=\"noreferrer noopener\">Edit data in Excel<\/a> feature have limitations such as not supporting many-to-many relationships and limited support for multiple choice data types.<\/p>\n\n\n\n<p>To overcome the many-to-many import problems, I developed a Python program to connect to Dataverse using the Web API and bind oData records together in a many-to-many relationships.<\/p>\n\n\n\n<p>From this I developed a wider library of methods that enable seamless interaction with Dataverse in Python. The complete code is available on <a href=\"https:\/\/github.com\/YesWeCandrew\/PyConnectDataverse\" data-type=\"URL\" data-id=\"https:\/\/github.com\/YesWeCandrew\/PyConnectDataverse\" target=\"_blank\" rel=\"noreferrer noopener\">GitHub<\/a>.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"connecting-to-dataverse-with-python\">Connecting to Dataverse with Python<\/h2>\n\n\n\n<p>The primary challenge with using Python to connect with Dataverse is that the existing <a href=\"https:\/\/pydataverse.readthedocs.io\/en\/latest\/index.html\" data-type=\"URL\" data-id=\"https:\/\/pydataverse.readthedocs.io\/en\/latest\/index.html\" target=\"_blank\" rel=\"noreferrer noopener\">Python library<\/a> is no longer maintained, and with the <a href=\"https:\/\/techcommunity.microsoft.com\/t5\/microsoft-entra-azure-ad-blog\/update-your-applications-to-use-microsoft-authentication-library\/ba-p\/1257363\" data-type=\"URL\" data-id=\"https:\/\/techcommunity.microsoft.com\/t5\/microsoft-entra-azure-ad-blog\/update-your-applications-to-use-microsoft-authentication-library\/ba-p\/1257363\" target=\"_blank\" rel=\"noreferrer noopener\">depreciation of ADAL<\/a> there are limited instructions on how to get the oAuth token needed to authenticate yourself.<\/p>\n\n\n\n<p>To address this, my program uses Azure Active Directory interactive logins and the MSAL library to establish an authenticated session with the <a href=\"https:\/\/learn.microsoft.com\/en-us\/power-apps\/developer\/data-platform\/webapi\/overview\" data-type=\"URL\" data-id=\"https:\/\/learn.microsoft.com\/en-us\/power-apps\/developer\/data-platform\/webapi\/overview\" target=\"_blank\" rel=\"noreferrer noopener\">Dataverse web API<\/a>. This session can then be leveraged to execute any of the numerous operations support by the API.<\/p>\n\n\n\n<p>The setup process is quick and straightforward, ensuring both convenience and enhanced security by avoiding the use of client keys that can lead to compromise. Additionally, it allows authentication with existing accounts, eliminating the need to add an application user to your environment.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"key-uses\">Key uses<\/h2>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Securely accessing all functions of the <a href=\"https:\/\/learn.microsoft.com\/en-us\/power-apps\/developer\/data-platform\/webapi\/query-data-web-api\" target=\"_blank\" rel=\"noreferrer noopener\">OData Dataverse API<\/a> OData Dataverse API with an authenticated <a href=\"https:\/\/www.geeksforgeeks.org\/session-objects-python-requests\/\" target=\"_blank\" data-type=\"URL\" data-id=\"https:\/\/www.geeksforgeeks.org\/session-objects-python-requests\/\" rel=\"noreferrer noopener\">Python session<\/a>. <\/li>\n\n\n\n<li>Downloading Dynamics 365 data using Python for efficient analysis.<\/li>\n\n\n\n<li>Adding many-to-many relationship data to Dataverse.<\/li>\n\n\n\n<li>Automating tedious manual processes such as populating option sets and choices.<\/li>\n\n\n\n<li>Anything you can think of using the OData API and Python!<\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"step-by-step-guide-to-connect-to-dataverse\">Step-by-step guide to connect to Dataverse<\/h2>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"step-1-create-a-new-azure-active-directory-app\">Step 1: Create a new Azure Active Directory app<\/h3>\n\n\n\n<div class=\"wp-block-columns is-layout-flex wp-container-core-columns-is-layout-1 wp-block-columns-is-layout-flex\">\n<div class=\"wp-block-column is-layout-flow wp-block-column-is-layout-flow\">\n<p>To begin, navigate to the <a href=\"https:\/\/aad.portal.azure.com\/#view\/Microsoft_AAD_IAM\/ActiveDirectoryMenuBlade\/~\/RegisteredApps\" data-type=\"URL\" data-id=\"https:\/\/aad.portal.azure.com\/#view\/Microsoft_AAD_IAM\/ActiveDirectoryMenuBlade\/~\/RegisteredApps\" target=\"_blank\" rel=\"noreferrer noopener\">Azure Active Directory portal<\/a> and create a new app, by clicking <strong>Azure Active Directory<\/strong> > <strong>App Registrations<\/strong> > <strong>+ New Registration<\/strong>.<\/p>\n\n\n\n<p>Give your app a name and select the supported account types. In most cases, choosing <strong>Accounts in this organizational directory only<\/strong> is recommended. Leave redirect URI blank for now and complete the registration process by clicking <strong>Register<\/strong>.<\/p>\n<\/div>\n\n\n\n<div class=\"wp-block-column is-layout-flow wp-block-column-is-layout-flow\">\n<figure class=\"wp-block-image aligncenter size-large is-resized\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/andyhowes.local\/wp-content\/uploads\/2023\/07\/AAD-App-registrations-477x1024.png\" alt=\"\" class=\"wp-image-543\" width=\"164\" height=\"354\"\/><\/figure>\n<\/div>\n<\/div>\n\n\n\n<figure class=\"wp-block-image aligncenter size-large is-resized\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/andyhowes.local\/wp-content\/uploads\/2023\/07\/Create-a-new-app-1024x871.png\" alt=\"\" class=\"wp-image-544\" width=\"512\" height=\"436\" srcset=\"https:\/\/andyhowes.local\/wp-content\/uploads\/2023\/07\/Create-a-new-app-1024x871.png 1024w, https:\/\/andyhowes.local\/wp-content\/uploads\/2023\/07\/Create-a-new-app-300x255.png 300w, https:\/\/andyhowes.local\/wp-content\/uploads\/2023\/07\/Create-a-new-app-768x653.png 768w, https:\/\/andyhowes.local\/wp-content\/uploads\/2023\/07\/Create-a-new-app-1536x1306.png 1536w, https:\/\/andyhowes.local\/wp-content\/uploads\/2023\/07\/Create-a-new-app.png 1830w\" sizes=\"(max-width: 512px) 100vw, 512px\" \/><\/figure>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"step-2-grant-app-permissions-for-dynamics-crm\">Step 2: Grant app permissions for Dynamics CRM<\/h3>\n\n\n\n<p>In the sidebar of the newly created app, select <strong>API permissions<\/strong> &gt; <strong>+ Add a permission<\/strong>.<\/p>\n\n\n\n<p>Locate the<strong> Dynamics CRM<\/strong> permission and select it, enabling <strong>user_impersonation<\/strong>.<\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter size-full is-resized\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/andyhowes.local\/wp-content\/uploads\/2023\/07\/Dynamics-CRM-permissions.png\" alt=\"\" class=\"wp-image-548\" width=\"213\" height=\"132\" srcset=\"https:\/\/andyhowes.local\/wp-content\/uploads\/2023\/07\/Dynamics-CRM-permissions.png 502w, https:\/\/andyhowes.local\/wp-content\/uploads\/2023\/07\/Dynamics-CRM-permissions-300x185.png 300w\" sizes=\"(max-width: 213px) 100vw, 213px\" \/><\/figure>\n\n\n\n<p>Once this permission has been selected, <strong>Grant admin consent for your organisation<\/strong> to enable you to interact with the data in Python using the permissions of the logged in user.<\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter size-large is-resized\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/andyhowes.local\/wp-content\/uploads\/2023\/07\/Grant-permission-1024x308.png\" alt=\"\" class=\"wp-image-549\" width=\"512\" height=\"154\" srcset=\"https:\/\/andyhowes.local\/wp-content\/uploads\/2023\/07\/Grant-permission-1024x308.png 1024w, https:\/\/andyhowes.local\/wp-content\/uploads\/2023\/07\/Grant-permission-300x90.png 300w, https:\/\/andyhowes.local\/wp-content\/uploads\/2023\/07\/Grant-permission-768x231.png 768w, https:\/\/andyhowes.local\/wp-content\/uploads\/2023\/07\/Grant-permission-1536x463.png 1536w, https:\/\/andyhowes.local\/wp-content\/uploads\/2023\/07\/Grant-permission-2048x617.png 2048w\" sizes=\"(max-width: 512px) 100vw, 512px\" \/><\/figure>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"step-3-configure-authentication\">Step 3: Configure authentication<\/h3>\n\n\n\n<p>To enable interactive MSAL authentication, you need to add a Redirect URI of http:\/\/localhost.<\/p>\n\n\n\n<p>To do this, in the sidebar of the app, select <strong>Authentication<\/strong> &gt; <strong>+ Add a platform<\/strong> &gt; <strong>Mobile and desktop applications<\/strong> &gt; Add the url <strong>http:\/\/localhost<\/strong><\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter size-large is-resized\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/andyhowes.local\/wp-content\/uploads\/2023\/07\/Add-redirect-uri-1-1024x473.png\" alt=\"\" class=\"wp-image-562\" width=\"512\" height=\"237\" srcset=\"https:\/\/andyhowes.local\/wp-content\/uploads\/2023\/07\/Add-redirect-uri-1-1024x473.png 1024w, https:\/\/andyhowes.local\/wp-content\/uploads\/2023\/07\/Add-redirect-uri-1-300x138.png 300w, https:\/\/andyhowes.local\/wp-content\/uploads\/2023\/07\/Add-redirect-uri-1-768x355.png 768w, https:\/\/andyhowes.local\/wp-content\/uploads\/2023\/07\/Add-redirect-uri-1-1536x709.png 1536w, https:\/\/andyhowes.local\/wp-content\/uploads\/2023\/07\/Add-redirect-uri-1.png 2032w\" sizes=\"(max-width: 512px) 100vw, 512px\" \/><\/figure>\n\n\n\n<p>Scroll down to <strong>Advanced settings<\/strong> and select Yes to <strong>Allow public client flows<\/strong>.<\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter size-large is-resized\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/andyhowes.local\/wp-content\/uploads\/2023\/07\/Allow-public-client-flows-1024x368.png\" alt=\"\" class=\"wp-image-553\" width=\"512\" height=\"184\" srcset=\"https:\/\/andyhowes.local\/wp-content\/uploads\/2023\/07\/Allow-public-client-flows-1024x368.png 1024w, https:\/\/andyhowes.local\/wp-content\/uploads\/2023\/07\/Allow-public-client-flows-300x108.png 300w, https:\/\/andyhowes.local\/wp-content\/uploads\/2023\/07\/Allow-public-client-flows-768x276.png 768w, https:\/\/andyhowes.local\/wp-content\/uploads\/2023\/07\/Allow-public-client-flows.png 1524w\" sizes=\"(max-width: 512px) 100vw, 512px\" \/><\/figure>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"step-4-get-app-information\">Step 4: Get app information<\/h3>\n\n\n\n<p>The app configuration is now complete. In the sidebar, click on <strong>Overview<\/strong> in the app sidebar and note the following details:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Application (client) ID<\/li>\n\n\n\n<li>Directory (tenant) ID<\/li>\n<\/ul>\n\n\n\n<p>Additionally, retrieve the base URL of your Environment. You can find this in the <a href=\"https:\/\/admin.powerplatform.microsoft.com\/environments\/\" data-type=\"URL\" data-id=\"https:\/\/admin.powerplatform.microsoft.com\/environments\/\" target=\"_blank\" rel=\"noreferrer noopener\">Power Platform Admin Centre<\/a> by selecting the environment you want to connect to.<\/p>\n\n\n\n<p><strong><strong>Always use a Development environment for testing, before using the API on your Production environment.<\/strong><\/strong><\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter size-large is-resized\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/andyhowes.local\/wp-content\/uploads\/2023\/07\/Find-environment-URL-1024x311.png\" alt=\"\" class=\"wp-image-555\" width=\"768\" height=\"233\" srcset=\"https:\/\/andyhowes.local\/wp-content\/uploads\/2023\/07\/Find-environment-URL-1024x311.png 1024w, https:\/\/andyhowes.local\/wp-content\/uploads\/2023\/07\/Find-environment-URL-300x91.png 300w, https:\/\/andyhowes.local\/wp-content\/uploads\/2023\/07\/Find-environment-URL-768x233.png 768w, https:\/\/andyhowes.local\/wp-content\/uploads\/2023\/07\/Find-environment-URL-1536x466.png 1536w, https:\/\/andyhowes.local\/wp-content\/uploads\/2023\/07\/Find-environment-URL-2048x622.png 2048w\" sizes=\"(max-width: 768px) 100vw, 768px\" \/><\/figure>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"step-5-clone-the-repo\">Step 5: Clone the repo<\/h3>\n\n\n\n<p>Clone the PyConnectDataverse repository from Github at: <a href=\"https:\/\/github.com\/YesWeCandrew\/PyConnectDataverse\" target=\"_blank\" rel=\"noreferrer noopener\">https:\/\/github.com\/YesWeCandrew\/PyConnectDataverse<\/a><\/p>\n\n\n\n<p>Use pip to install any packages you haven&#8217;t installed previously, such as MSAL.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"step-6-configure-your-environment-json\">Step 6: Configure your environment JSON<\/h3>\n\n\n\n<p>Using the information gathered in step 4, create an environment JSON. Copy the <strong>example-env.json<\/strong> file and replace the values with the details of your app and environment.<\/p>\n\n\n\n<p><strong>Always use a Development environment for testing first, before using the API on your Production environment.<\/strong><\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"step-7-use-your-favourite-ide-to-interact-with-dataverse-with-python\">Step 7: Use your favourite IDE to interact with Dataverse with Python&nbsp;<\/h3>\n\n\n\n<p>Test the connection by configuring the parameters in the pcd_test_connection.py file. Run the module to ensure a connection is established. A window will appear, prompting you to log in. If you have already logged in, the process will be automated.<\/p>\n\n\n\n<p>You can now utilise the provided pcd Python files or develop your own functions to interact with Dataverse using Python. Remember to update the specified parameters in each file accordingly.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"py-connect-dataverse\">PyConnectDataverse<\/h2>\n\n\n\n<p>Enjoy the benefits of Python-based interaction with Dataverse!  I keep finding new uses for this program and am keen to keep using it to speed up common Dataverse tasks.<\/p>\n\n\n\n<p>As new uses for this program continue to emerge, I encourage you to contribute by forking the repository and submitting pull requests to have your useful functions added. I am open to additions and suggestions to enhance the repository further.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Discover the power of connecting to Dataverse, Microsoft Dynamics and Power Apps with a Python program I developed.<\/p>\n","protected":false},"author":1,"featured_media":559,"comment_status":"closed","ping_status":"closed","sticky":true,"template":"","format":"standard","meta":{"rank_math_lock_modified_date":false,"footnotes":""},"categories":[1],"tags":[49,40,39,46,50,47,48],"class_list":["post-540","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-general","tag-data-management","tag-dataverse","tag-dynamics","tag-microsoft-dynamics","tag-programming","tag-python","tag-web-api"],"_links":{"self":[{"href":"https:\/\/andyhowes.local\/wp-json\/wp\/v2\/posts\/540"}],"collection":[{"href":"https:\/\/andyhowes.local\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/andyhowes.local\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/andyhowes.local\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/andyhowes.local\/wp-json\/wp\/v2\/comments?post=540"}],"version-history":[{"count":13,"href":"https:\/\/andyhowes.local\/wp-json\/wp\/v2\/posts\/540\/revisions"}],"predecessor-version":[{"id":563,"href":"https:\/\/andyhowes.local\/wp-json\/wp\/v2\/posts\/540\/revisions\/563"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/andyhowes.local\/wp-json\/wp\/v2\/media\/559"}],"wp:attachment":[{"href":"https:\/\/andyhowes.local\/wp-json\/wp\/v2\/media?parent=540"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/andyhowes.local\/wp-json\/wp\/v2\/categories?post=540"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/andyhowes.local\/wp-json\/wp\/v2\/tags?post=540"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}