Integrating Azure Application Insights with Microsoft Copilot Studio for Enhanced Monitoring and Error Logging

Table of Contents

• Introduction
• Why Use Application Insights with Copilot Studio?
• Step-by-Step Guide to Integrating Application Insights with Copilot Studio
  • Step 1: Setting Up Application Insights
  • Step 2: Configure the Copilot
  • Test the Runtime Error Events
  • Test 2: System / Logical Errors
  • Test 3: Custom Telemetry Events
• Copilot Audit Logs
• Conclusion

Introduction

Microsoft Copilot Studio offers an innovative way to build interactive conversational bots. However, to ensure these bots operate smoothly and meet user expectations, integrating effective monitoring tools is crucial. This is where Azure Application Insights comes into play. Application Insights, a component of Azure Monitor, provides deep insights into application performance, user behaviors, and error diagnostics, making it an invaluable tool for enhancing the monitoring and error logging capabilities of your Copilot bots.

In this blog, we will explore how to integrate Application Insights with Copilot Studio Copilots to improve error logging and monitoring, ultimately leading to more reliable and user-friendly interactions.

Why Use Application Insights with Copilot Studio?

Application Insights offers a range of features that are beneficial for monitoring and improving Copilot performance:

1. Real-Time Monitoring: Track the real-time performance and health of all your Copilots in the Copilot Studio, identifying issues as they arise.
2. Detailed Error Logging: Capture detailed logs of errors and exceptions, enabling quicker diagnostics and resolution.
3. User Interaction Analytics: Analyze user interactions to understand usage patterns and identify potential improvements.
4. Custom Telemetry: Collect and analyze custom events tailored to your specific needs.

Step-by-Step Guide to Integrating Application Insights with Copilot Studio

Step 1: Setting Up Application Insights

1. Head over to Azure Application Insights

2. Specify the Subscription and instance details and click on Review + Create

3. Copy the connection string from the overview page

4. View the captured Copilot events in the events tab of the side pane

Step 2: Configure the Copilot

1. Select one of the Copilots created and go to Settings

2. To connect the Copilot with Application Insights:

   • Select Copilot details
   • Click on Advanced
   • Paste the connection string from Application Insights
   • Select the type of events and activities to capture
   • Click on Save

Test the Runtime Error Events

1. Simulate a division by zero error in Power Fx by adding the formula Value(9)/Value(0) in the message node

2. Test the Copilot in the test pane to see the runtime division by zero error

3. Monitor these events in Application Insights
   • View more insights to see detailed error logs
4. Click on On Error Log to see detailed error logs

Test 2: System / Logical Errors

1. Induce a system error by self-calling the topic, resulting in an infinite loop
   • From within the topic named Tester, add a node and select Topic Management -> Go to another topic -> Tester

2. Test the Copilot from the test pane to see the infinite loop error

3. View the detailed error logs in Application Insights

Test 3: Custom Telemetry Events

1. Log custom events using the “Log a custom telemetry event” action

2. Enter the custom event details to be logged

3. Invoke the custom telemetry node and check the logs in Application Insights

Copilot Audit Logs

The events around Copilot, such as creation, deletion, changes in authentication, basic details, and topic creation, are also monitored and written into audit logs accessible from Microsoft Purview. The general events logged are listed below:

1. Access these logs from the compliance portal by clicking on the Audit tab

2. Search for specific events within a timeline

3. View the detailed list of selected search events

Conclusion

Integrating Azure Application Insights with Microsoft Copilot Studio enhances monitoring, diagnosing, and optimizing Copilot applications. It provides detailed telemetry and error logging to gain insights into performance, quickly resolve issues, and understand user interactions. By capturing critical data on runtime and system errors and logging custom telemetry events, Application Insights ensures robust Copilot bots. Coupled with audit logs in Microsoft Purview, it ensures comprehensive tracking of key events for effective governance, reliability, and security of Copilot solutions.