Advisories

Researchers from ReversingLabs have identified malicious Python packages located on the popular Python package repository “Python Package Index (PyPI)” posing as a software development kit (SDK) from SentinelOne.  The package mimics the legitimate SDK that's offered by SentinelOne to its customers but adds backdoor and data exfiltration features. 

The full article that includes the writeup and IOCs ( Data exfiltration IPs and package SHA1 hashes) can be found at this link, and the ConnectWise Security Operations Team has been provided the following information from SentinelOne: 

"SentinelOne is aware of the report from Reversing Labs regarding malicious packages uploaded to the PyPi (Python Package Index) repository misrepresenting themselves as SentinelOne SDK. 

A malicious Python package was first uploaded to PyPi on Dec 11, 2022, and as of Dec 13, 2022, the package had been updated 20 times. The report advises that the package contains a malicious backdoor with a programmatic delay before activation. We have confirmed that our customers are safe and have not seen any evidence of compromised clients due to this incident. 

Packages posting as legitimate software and leveraging the PyPi repository are becoming more common and are part of a trend toward integrating threats into software supply chains and development pipelines. 

We recommend only using SDK packages provided through the SentinelOne management console. 

PyPI has removed the malicious package, and we are working to investigate further." 

As an industry best practice, ConnectWise recommends partners download content (SDKs, executables, installation packages, etc.) directly from the vendor to minimize risk and always verify script content prior to execution.  

Vulnerability Type: Time-of-check Time-of-use (TOCTOU) Race Condition  

Vulnerability Details 
SafeBreach Labs researcher Or Yair uncovered vulnerabilities in several leading EDR and AV solutions, including SentinelOne, that allows a non-privileged user to create NTFS reparse points, which creates a path that “links” to a different path. The SentinelOne agent uses Windows functionality to get a path of a file to mitigate. A malicious actor may replace the path with a different path to a file to which it does not have privileges. This can potentially turn the agent into a malicious data wiper.  

Products Impacted  
Microsoft Windows with SentinelOne agents running all versions prior to 22.2.4.558 are vulnerable.  

SentinelOne agents are utilized in the following ConnectWise products: ConnectWise SentinelOne Control, ConnectWise SentinelOne Complete, ConnectWise MDR with SentinelOne, and ConnectWise MDR Premium with SentinelOne. 

This exploit was also tested against Defender, Defender for Endpoint, TrendMicro Apex One, Avast Antivirus, and AVG Antivirus and was found to be vulnerable.  

Mitigation 
In order to be protected, you are required to install the latest SentinelOne policy override in version 22.2 SP1 (22.2.4.558) on your Windows agent endpoints. ConnectWise SOC teams have already updated all the ConnectWise SentinelOne EDR and MDR consoles with the 22.2.4.558 agent. 

After the updates have been deployed, please verify in the SentinelOne console if your machine has a pending reboot that needs to be actioned in case this is required to complete the installation.  

If you have any questions about the updating process, please contact our security support teams at securitypartnersupport@connectwise.com.   

We are aware of a phishing campaign that mimics ConnectWise Control New Login Alert emails and has the potential to lead to unauthorized access to legitimate Control instances. We know email phishing attacks continue to get more sophisticated, mirroring legitimate email and web content.

A sample of this phishing email is shown in the screenshot below and contains a “click here” link to a malicious site. ConnectWise has issued take-down requests for the malicious site and domains.

If you are concerned that you may have been compromised, please follow the steps in this security alert checklist. We also recommend reviewing the Control security guide and best practices for further securing your instance, as well as verifying that links, your account ID, and your domain are accurate.

Of note, Control does send legitimate New Login Alerts via email as shown in this screenshot.  The legitimate “click here” link references the aforementioned security alert checklist that exists as a knowledge base article on our site.

This is a more sophisticated attempt – some of the standard phishing attack indicators aren’t there, like misplaced graphics, or spelling inconsistencies. We encourage our partners to stay vigilant in looking for clues to avoid mistakenly clicking on nefarious content. Before clicking, make sure content reflects:

• Email domains owned by trusted sources
• Links that go to places you recognize

If you have questions, suspect you received a phishing attempt, or need to report a security or privacy incident, please visit our ConnectWise Trust Center. You can report both a non-active security incident, report a security vulnerability, or call our Partner InfoSec Hotline at 1-888-WISE911.

We want to provide reminders to our partners about email security best practices.  

Phishing remains a significant attack vector fronting attack chains in some very high-profile security incidents.  As such, it is imperative that organizations implement email security controls to prevent impersonation/spoofing of their users and domains.  SPF, DKIM, and DMARC provide a layer of protection against this by working in tandem to authenticate email and helping to ensure that the sender REALLY is who they say they are.   

SPF, DKIM, and DMARC Defined   

• SPF (Sender Policy Framework) is an email validation protocol designed to detect and block email spoofing. It allows mail exchangers to verify that incoming mail from a specific domain comes from an IP Address authorized by that domain’s administrators.  
• DKIM (DomainKeys Identified Mail) utilizes cryptographic signatures by which mail service providers can verify the authenticity of the sender.  
• DMARC (Domain-based Message Authentication, Reporting & Conformance) aligns the SPF and DKIM mechanisms and allows organizations to apply policies regarding unauthorized use of email domains. 

For more information and details on how to setup/configure SPF/DKIM/DMARC, there are several good resources available including the following:   

SPF: https://www.proofpoint.com/us/threat-reference/spf 

DKIM: https://www.proofpoint.com/us/threat-reference/dkim 

DMARC: https://www.proofpoint.com/us/threat-reference/dmarc 

Security is a top priority at ConnectWise. Our primary goal is to provide robust, secure products and services to our partners. We also acknowledge that no technology is perfect, and ConnectWise believes that working with skilled security researchers and partners across the globe is crucial in identifying weaknesses in any technology. If you believe you've found a security issue in our product or service, we encourage you to notify us via our Vulnerability Disclosure Program. We welcome working with you to resolve the issue promptly.  

We are proud to be part of a community that remains equally committed to secure practices. 

We apologize to our partners for the disruption in service last week pertaining to our virtual community. It is now online, and our product and other teams look forward to engaging with you.

Like many ConnectWise experiences (e.g. our University) our virtual community platform leverages SSO to authenticate users and ensure only authorized partners engage in our community. Our SSO mechanism did its job—only allowing verified ConnectWise partners to register, accept the terms and conditions and use the virtual community platform. There was no malicious attack on our SSO capabilities.

Last week, a valued partner (via our VDP and respected admins of the MSPGeek community) raised concern about information our virtual community search was displaying to registered community member partners. Directory search was working as intended in most cases, but a configuration issue was allowing non-registered partners to be returned in a search. This information included "first name", "last name", "company name" (and in some cases, "business title"). Although this information can easily be obtained via other platforms (like LinkedIn), it raised understandable partner concern.  Only 15 registered partner members conducted searches since the community launch, and while we were unable to validate the results of their searches due to a limitation in our vendor’s API, we do know that only 18 non-registered partners "profiles" were viewed by registered partner members as a result of those searches.

We remediated this issue within hours but took the site down pending a full review in accordance with our InfoSec policy. No malicious activity was discovered, no data was lost, and this triggered no data privacy actions in the jurisdictions involved.

Although a common community feature, partners also expressed concern that a registered partner community member could conduct a search by "company name". We understand it is important for partner employees (registered users) to determine how much or how little information is shared with others in the virtual community. Here’s what we did:

• We reconfigured the virtual community to—after authentication—consume only basic information about registered users of the virtual community who accept the terms of service.
• Default settings now limit directory search fields to first name and last name.
• Member directory is “on” for registered partner member viewing to help deliver the experience TSPs expect when joining a virtual community. However, we have set default privacy settings for all registered members such that only their first name, last name (and profile photo where uploaded) will display when being searched for by members who aren’t their approved contacts.
• Registered members may proactively change the privacy settings associated with their user profile to control the level of information that is shared with approved contacts or other members. Partners can find more information about privacy settings in the Virtual Community FAQs.

As a courtesy, we are notifying the 18 individuals mentioned above and are reaching out to the 15 partners who conducted searches to gain their assurance this information will not be used beyond community networking.

Finally, we know it is important to you to hear what we learned from this. Our beta testing (both internal and with partners) in the 30 days prior did not expose this configuration issue. This taught us about extra measures we can and will take in the future; and we have immediately implemented additional multi-layered testing and QC mechanisms to our processes.  

Transparency on all sides benefits our community. We want to thank the partner who reported this, and the partners who collaborated with us on this issue. If you have additional questions about this matter, please contact security@connectwise.com.

Although directory functionality for our virtual community platform was disabled when we launched our community, an issue with our third-party platform’s configuration was discovered. This issue allowed partner first name, last name, and company name (and in some cases, job title) to be returned in the search. We remediated this issue but shut the web site down in an abundance of caution so we could conduct a full assessment in compliance with our InfoSec protocols. To be clear, no malicious activity has been discovered. More specifically, our analysis shows that only partners and ConnectWise employees conducted this search since our community was launched—less than 20 partners searched and many searches were this morning from partners who were helping us test this issue. We have been able to track every search to a legitimate user. We have consulted with our legal counsel, and this has not triggered any GDPR issues. We will share more with our partners when we have more details as our investigation continues.

A potential issue with the virtual community site is being assessed. As a precautionary measure, we have temporarily put the site in maintenance mode while we continue our investigation. To be clear, no malicious activity has been identified. We will update partners shortly. 

As mentioned yesterday, we released a patch for Manage versions 2021.2 and 2021.3 that will safely re-enable the Global Search capability once installed. Today, a patch was released for Manage versions 2020.4 and 2021.1 that will safely re-enable Global Search.

To install this patch, please follow the instructions via this link: https://docs.connectwise.com/ConnectWise_Support_Wiki/System/Manage_On_Premise_-_Log4J_remediation 

Manage partners: If you have any questions related to this patch, please contact our Support team at help@connectwise.com.  

All partners: Your security remains our top priority. If you have any security-related questions or concerns, please contact security@connectwise.com. 

We appreciate your continued partnership.  

Thank you,
The ConnectWise InfoSec Team

A new patch that will safely re-enable the Global Search capability for Manage is now available for all Manage on-premise partners on versions 2021.2 and 2021.3. If you are not using version 2021.2 or 2021.3, we ask that you please continue to keep Global Search disabled for security purposes. Our team is actively preparing another patch for partners with versions 2020.4 and 2021.1 and we will provide another update when it is available. 

To install this patch, please follow the instructions via this link: https://docs.connectwise.com/ConnectWise_Support_Wiki/System/Manage_On_Premise_-_Log4J_remediation  

Manage partners: If you have any questions related to this patch, please contact our Support team at help@connectwise.com.  

All partners: Your security remains our top priority. If you have any security-related questions or concerns, please contact security@connectwise.com. 

We appreciate your continued partnership.  

Thank you,
The ConnectWise InfoSec Team

Global Search Update for ConnectWise PSA On-Premise Partners: As of today, December 21, we are pleased to share that SOLR has finished publishing an updated fix. Our Development Team has reviewed the update and is currently testing the script. As soon as the fix has been tested successfully, we will release it to all Manage on-premise partners through a patch. Partners will then be able to install the patch through their Updater. Once the patch is installed, Global Search capability will be re-enabled. Please stay tuned for another update this week which will include steps to install the patch. Thank you for your patience and flexibility.  

As always, please reach out to Security@ConnectWise.com to report a security issue with ConnectWise products.  

We appreciate your continued partnership.  

Thank you,
The ConnectWise InfoSec Team

As you are aware, over the weekend the Apache Software Foundation released version 2.17.0 of Log4j to address a new denial of service vulnerability. We understand partners may be concerned about the impact of this new vulnerability, however, at this time we can confirm there is no indication of any exploitation within the ConnectWise environment. Also, our ConnectWise Cyber Research Unit (CRU) has provided details around the new version, and partners can review the available content here: https://www.connectwise.com/resources/a-new-new-new-new-log4j-vulnerability

Moving forward, we are incorporating this new information into our work to ensure ongoing protection for all our partners, products and services.  

In addition, we are providing an update via email to our Perch partners regarding the new vulnerability.  

Please reach out to Security@ConnectWise.com with any additional security questions or to report a security issue. We appreciate your continued partnership.  

Thank you,
The ConnectWise InfoSec Team

Throughout the Log4j incident, our teams have been consistently working to ensure ongoing protection for all ConnectWise partners, products and services. With that, we have developed two new solutions to help our ConnectWise Automate, Command, and RMM partners detect any potential Log4j vulnerabilities in their systems. 

For ConnectWise Automate Partners 

Our ConnectWise Automate team has added a new release of a “Log4j Windows Vulnerability Check” Solution within the Automate Solution Center. Partners may now download the new solution by following the steps below: 

• Restart the Solution Center Server on your Automate server to force the reload of Solution Center data. 
• Once the Solution Center has restarted, the Log4j Windows Vulnerability Check Solution will be available for install under the Security Category. 
• The Solution adds a new Script “log4j Windows Vulnerability Check” located in the Maintenance > Patching folder.  When run against Windows endpoints, the script will search all local files looking for .jar/.war/.ear files containing potentially vulnerable versions of Log4J. If vulnerable files are found, a ticket will be created for the system with the list of potentially vulnerable files. 
• If you have any questions related to this new solution, please contact help@connectwise.com. 

For ConnectWise Command & ConnectWise RMM Partners 

Our ConnectWise Command and RMM teams have provisioned a new capability within both products that help partners automatically detect any potential Log4j vulnerabilities. To utilize this new capability, please follow the steps below: 

• In your instance, visit Automation > Task, and search for “Detect Log4j Vulnerabilities”.  
• Select the schedule option to schedule the Task to run against your target systems.  
• The Task output will return the full file path of any potentially vulnerable file when it is run against Windows endpoints. 
• If you have any questions regarding this capability, please open a Support Ticket within your ITSupport Portal. 

As always, please reach out to Security@ConnectWise.com to report a security issue with ConnectWise products. We appreciate your continued partnership.  

Thank you, 

The ConnectWise InfoSec Team

We have no new issues to report at this time. We will provide another update this evening (ET).  

Please continue reaching out to Security@ConnectWise.com with any additional questions or to report an issue. We appreciate your continued partnership.  

Thank you, 

The ConnectWise InfoSec Team 

As previously communicated, our team discovered last week that Manage on-premise Global Search capability had a third-party component that is impacted by the Log4j vulnerability. We immediately provided partners with procedures to terminate this service to reduce any potential security risk until a patch is deployed.  

However, we understand the impact disabling this capability has on your business and that it may potentially cause performance degradation within Manage. In order to improve your server performance while our third-party threat intelligence and forensics partners continue to work to remediate any issues, we recommend partners complete these updated instructions in this documentation: https://docs.connectwise.com/ConnectWise_Unified_Product/Supportability_and_Vulnerability_Statements_for_ConnectWise_Unified_Product/How_to_Disable_the_ConnectWise_Global_Search. Please ensure you are logged in to the University via ConnectWise SSO to view these steps. 

As always, please reach out to Security@ConnectWise.com with any additional questions or to report an issue. We appreciate your continued partnership.  

Thank you, 
The ConnectWise InfoSec Team 

After a comprehensive review to validate no vendor exposure and to confirm that no exploitation was observed, we re-enabled purchase capabilities of our Marketplace and global search capability of Manage Cloud. Partners can once again use these features. Please be aware that Manage on-premise Global Search capability remains suspended, and we will provide an update when it can be safely re-enabled. 

Please reach out to Security@ConnectWise.com with any additional questions or to report an issue. 

Thank you for your continued partnership, 
The ConnectWise InfoSec Team 

We appreciate your patience as our teams continue their work to investigate and remediate any issues caused by the Log4j vulnerability. As previously communicated, no new threats have been identified by ConnectWise beyond what was reported in our Trust Center updates earlier this week. 

At this time, the status of all products and services remains the same, and our third-party threat intelligence and forensic partner’s work consistently reflects no new discoveries of concern.  We will provide another update tomorrow. 

As always, please reach out to Security@ConnectWise.com with any additional questions or to report an issue.  

Thank you for your continued partnership, 
The ConnectWise InfoSec Team 

Our work to investigate and remediate any issues caused by the Log4j vulnerability continues. Although still underway, our third-party threat intelligence and forensic partner’s work continues to reflect no new discoveries of concern. In addition, no new threats have been identified by ConnectWise beyond what was reported in our earlier Trust Center updates.  

As previously communicated, we are working with our (Invent) Marketplace partners to ensure there is no vendor exposure. However, if you use a third-party integration or plugin to our solutions, we ask that you follow best practice for such situations and work with your vendor directly for questions or assistance in ensuring the security of those integrations. Also, if you have created your own private integrations or plugins, we ask that you take measures to ensure no exploitation or compromise.  It's important to note that although some integrations may not be directly compatible with Java or Log4j, the integrations can still call out to a service that is.

Also, as we are concluding our investigation into the Fortinet vulnerability that we previously reported, the majority of our StratoZen environment was back online this morning, but it is fully online as of tonight.   

Please reach out to Security@ConnectWise.com with any additional questions or to report an issue. We appreciate your continued partnership.

Thank you, 

The ConnectWise InfoSec Team 

We know that maintaining your business continuity is important—we thank you again for your patience as our teams work around the clock to investigate and remediate any issues caused by the global Log4j vulnerability. Doing everything we can to protect you and your customers remains our highest priority. No new threats have been identified by ConnectWise at this time beyond what was previously reported (included below for your convenience). Our third-party threat intelligence and forensics experts have made significant progress in their work to assess our ConnectWise environments, however, that work is still underway. Please continue to visit this page for the latest updates.

Current Status:

• One cloud service, Perch, had third-party components that were potentially vulnerable. This was remediated immediately on Friday, December 10. No exploitation has been observed.
• {Updated 12/13} On Friday, December 10 we notified Manage partners that ConnectWise PSA on-premise Global Search capability has a third-party component which is affected by this vulnerability. We provided Manage on-premise users with instructions to follow to terminate that service until we have remediated the situation. We are still working on this item and will update you when our work is complete. Thank you for your patience.
• {Updated 12/13} Although no exploitation was observed, we suspended purchase capabilities of our Marketplace and global search capability of Manage Cloud while we validate there is no vendor exposure. Our comprehensive review is still underway. Thank you for your patience.
• {Updated 12/13} On Saturday, December 11, 2021, we confirmed with third-party Fortinet that their FortiSIEM product, which is leveraged by our StratoZen solution, is vulnerable to the zero-day Log4j exploit and therefore a potential target. We temporarily restricted all network access to our hosted StratoZen servers over the weekend but have now restored most of the services. Our StratoZen partners received direct communication with more details. Any partners with self-hosted instances of StratoZen or public-facing components (who use FortiSIEM) who have not already done so should immediately take the steps outlined here. We also recommend if you have publicly accessible instances of FortiSIEM that are not protected by a VPN or other secure access method that you close down public access and review your system data for exposure.

Please reach out to Security@ConnectWise.com with any additional questions or to report an issue. Thank you for your patience as we and many companies around the world navigate this issue. We will do our utmost to conclude our work quickly. 

We appreciate your continued partnership.

Thank you,

The ConnectWise InfoSec Team

In follow up to our update posted last evening (see below), our third-party threat intelligence and forensic experts are still conducting their assessment. No new issues have been discovered at this time. We will provide our next update tomorrow morning ET.

Please reach out to Security@ConnectWise.com with any additional questions or to report an issue.

Thank you for your patience as we and many companies around the world navigate this issue. We will do our utmost to conclude our work quickly. 

Please refer to the following update in follow up to tonight’s previous post:

Our investigation of the Log4j vulnerability continues to ensure our partners are protected. We are presently working with our third-party vendors to confirm their status and any remediation plans, where appropriate. Out of an abundance of caution, while we engage with our partners on this review, we have taken the following steps: 

• One cloud service, Perch, had third-party components that were potentially vulnerable and were remediated immediately. No exploitation has been observed.  
• As we shared with Manage partners, Manage on-premise's Global Search capability has a third-party component which is affected by this vulnerability. Procedures to terminate that service were provided to Manage On-prem users until such time the third-party services could be remediated.   
• Although no exploitation was observed, we suspended purchase capabilities of our Marketplace and global search capability of Manage Cloud while we validate there is no vendor exposure. We will update partners via our Trust Center once it has been re-enabled.  
• {Update as of 8:00pm ET} At 4:00 PM ET on December 11, we restricted all network access to our StratoZen hosted environment as we investigated a potential third-party issue and notified our partners accordingly. This evening we confirmed with third-party Fortinet that their FortiSIEM product, which is leveraged by our StratoZen solution, is vulnerable to the zero-day log4j exploit and therefore a potential target. We are now taking steps outlined by Fortinet to remediate this in our hosted StratoZen environment--we will move as quickly as we can but expect this to take into tomorrow. We have sent instructions to all partners who are self-hosted to immediately take the steps outlined here if they use FortiSIEM. Our third-party threat intelligence and forensics experts are also assessing the situation to ensure no further action is required.

Please reach out to Security@ConnectWise.com with any additional questions or to report an issue. 

Our investigation of the Log4j vulnerability continues to ensure our partners are protected. We are presently working with our third-party vendors to confirm their status and any remediation plans, where appropriate. Out of an abundance of caution, while we engage with our partners on this review, we have taken the following steps: 

• One cloud service, Perch, had third-party components that were potentially vulnerable and were remediated immediately. No exploitation has been observed.  

• As we shared with Manage partners, Manage on-premise's Global Search capability has a third-party component which is affected by this vulnerability. Procedures to terminate that service were provided to Manage On-prem users until such time the third-party services could be remediated.   

• Although no exploitation was observed, we suspended purchase capabilities of our Marketplace and global search capability of Manage Cloud while we validate there is no vendor exposure. We will update partners via our Trust Center once it has been re-enabled.  

• At 4:00 PM ET, we restricted all network access to our StratoZen hosted environment as our team does a complete scan and evaluation.  

Please reach out to Security@ConnectWise.com with any additional questions or to report an issue. 

We are aware of Log4j vulnerability. There is no indication of any exploitation of this vulnerability. Our teams are actively reviewing the situation to determine any risk to our products or partners. We will provide updates as more information becomes available. Thank you for your patience.

If you are a ConnectWise PSA on-premises partner, we recommend you please login and review the detailed instructions here: https://docs.connectwise.com/ConnectWise_Business_Knowledge/300/How_to_Disable_the_ConnectWise_Global_Search 

Dear Partners, 

I specifically want to discuss four areas relevant to the Kaseya incident and the recently published guidance from the FBI and the Cybersecurity and Infrastructure Security Agency (CISA): Mandatory MFA, Admin Access Restrictions, Web Application Firewalls (WAF) and Removing Anti-Virus Exclusions. 

• Mandatory Multi-factor Authentication (MFA): Currently, all agent-based products have mandatory MFA. Several other products have MFA as a configurable option. We plan to move all products to a mandatory MFA model by the end of 2021 and will be soon rolling out resources, education, and communications to help our partners make this transition. 
• Restricting Access to Admin Interfaces via IP limitations: Today, ConnectWise Control supports IP restrictions. Automate, and all other products will implement IP restrictions by the end of Q3, 2021. 
• Web Application Firewall (WAF): This is under evaluation in Q3, 2021 for our various products to execute both with and without the IP limiting features. 
• Removing Anti-Virus exclusions: AV exclusions for all products will be eliminated by the end of Q3, 2021. 

Here are some additional practices and programs already launched: 

• SOC2 Type 2 Certification: All products are SOC2 Type 2 certified and are re-certified every six months. 
• Cloud Environment Monitoring: Product cloud environments are monitored 24/7 by our SOC for suspicious/malicious activity.  
• Vulnerability Management:  All products are subject to multiple security assessments including automated testing in the delivery pipeline, internal red-teaming, external penetration tests, and Bug Bounty. 
• Malware Protection: Cloud infrastructure is protected using advanced endpoint detection and response capabilities.  
• Delivery Pipeline: ConnectWise subjects its development and delivery pipeline to threat modeling to improve security against supply chain attacks.  
• Disaster Recovery: Data backup and disaster recovery programs are in place across all cloud environments. Access and encryption controls are established to safeguard data back-ups.  All recovery and data restoration plans are tested and updated regularly. 

Cyber threats are ever present and evolving, and we are committed to not only delivering best practices within our products, but also keeping you up to date on our progress and resources. I encourage you to look at the other pages on our Trust Center for information regarding how we secure our environments, request/view our SOC2 and SOC3 reports, sign up to receive our security bulletins, and more. 

As always, if you need to report an incident or vulnerability within our products, you can also do that through our Trust Center or by contacting security@connectwise.com.  

Thank you for your partnership. 

Tom Greco 

CISO, ConnectWise 

Dear Partners,

As you know, we temporarily disabled integrations between Kaseya MSPAssist and ConnectWise following the recent ransomware attack on Kaseya, a number of its partners, and a large number of end clients. Shortly after the attack, Kaseya hired Mandiant, whose forensics report confirmed the attack on VSA.

On July 14, we received additional information from Kaseya allowing us to assess any residual risk in the MSPAssist environment and we have determined that we will re-enable the integration into ConnectWise PSA and Automate. 

To ensure you have had time to prepare, we will re-enable this tomorrow, July 16 at 10am ET.

We understand the business impact of this disabled integration and want to assure you that our top priority is always to ensure the security of our products and systems to protect you and our partner community from cybercrime. We are pleased that we were able to successfully work together with Kaseya to keep our mutual partners safe.   

As always, we urge our partners to take the following steps to manage their own risk with this and any integration:

• Assure that the credentials used for the integration are configured with the least privilege necessary to function. Do not implement with administrative level permissions.  Please contact Kaseya for instructions on configuring permissions.
•  Know how to disable the integration - or any integration - within your admin interface if you are still not comfortable with the integration being active.
  • To disable an integration,go to System > Members > API Keys and search for API Keys of an integration you wish to disable. Then navigate to that member > API Keys and delete the API Key for that integration. This will disable all integrations using those credentials.
  • It may be a good idea to also cycle all of the API Keys to ensure there are not unused Keys still active and old keys have not been shared with anyone.

Additionally, cybersecurity updates, resources, and information can always be found on our Trust Center and at www.connectwise.com/rapidresponse.

Thank you for your continued partnership. 

Sincerely,

Tom Greco

CISO, ConnectWise

Dear Partners, 

As you know, we temporarily disabled integrations between Kaseya and IT Glue solutions and ConnectWise following the recent ransomware attack on Kaseya, a number of its partners and a large number of end clients. Shortly after the attack, Kaseya hired Mandiant, whose forensics report confirmed the attack on VSA. Since July 2, we have been in communication with Kaseya. We let Kaseya know that once an accredited third-party confirmed the IT Glue environment was not impacted by the VSA incident, we would re-enable that integration.  

On Saturday, July 10, we received the first written Mandiant report referencing the IT Glue integration. After reviewing the statement provided by Mandiant and performing our own risk assessment, we have determined that we will re-enable the IT Glue integration into ConnectWise PSA and Automate. To ensure you have had time to prepare, we will re-enable this tomorrow, Tuesday, July 13, at 10:00am ET. We are pleased that we were able to successfully work together with Kaseya and IT Glue to keep our mutual partners safe.   

We understand the business impact of this disabled integration and want to assure you that our top priority is always to ensure the security of our products and systems to protect you and our partner community from cybercrime.  

As always, we urge our partners to prepare for managing their own risk with this and any integration with the following: 

• Assure that the credentials used for the integration are configured with the least privilege necessary to function. Do not implement with administrative level permissions. See documentation on credentials and permission levels here. 
   
•  Know how to disable this integration – or any integration – within your admin interface.  

• This is useful if you are still not comfortable with the integration being active. 
• Also, it is imperative to have a rapid response process in place, should there ever be an issue due to the integration. See documentation here on: Removing a PSA integration or Pausing a PSA sync.

Additionally, cybersecurity updates, resources, and information can always be here found on our Trust Center and at www.connectwise.com/rapidresponse.  

Thank you for your continued partnership.   

Sincerely,   

Tom Greco 

CISO, ConnectWise

Dear Partners, 

We have received some questions about when we will re-enable IT Glue/Kaseya integrations following the ransomware attack against Kaseya, which impacted some of our shared partners. Given the sophistication and scope of the attack, we temporarily disabled integrations between Kaseya platform products and ConnectWise.

We will re-enable the IT Glue integration (and others) once we officially confirm that there is no vulnerability or threat through third-party validation or through our own due diligence to confirm there is no risk to our partners as it relates to this incident. If it is confirmed that there was in fact a compromise of anything on the Kaseya or IT Glue side that integrates with ConnectWise applications, cybercriminals could, in certain situations, potentially leverage that to possibly exfiltrate data or execute code remotely. We engaged with Kaseya to ensure our concerns are not only heard but addressed, and currently the third-party validation provided confirms VSA’s exposure but did not indicate any analysis had been done for IT Glue or other Kaseya solutions. We’ve requested this from Kaseya/IT Glue and we have also offered to help fund such an audit.

We apologize for the delay, but our top priority continues to be ensuring our partners and your clients are protected. Thank you for your patience as we work through the fallout from the Kaseya attack. We will continue to provide you with regular updates. In the meantime, you can find resources here on the Trust Center and at https://www.connectwise.com/company/rapid-response.

Thank you for your partnership. 

Sincerely,   

Tom Greco 

CISO, ConnectWise

Be aware that there is currently a malware scam campaign attempting to take advantage of the recent Kaseya VSA ransomware attack.

• Anyone targeted by this campaign will receive an email with an attachment named “SecurityUpdates.exe.” Under no circumstances should anyone attempt to download this file.
• The email may also contain a link pretending to be an official security update from Microsoft designed to patch the Kaseya vulnerability.

Thank you, 

Tom Greco, Chief Information Security Office, ConnectWise

July 6, 2021: A Message from ConnectWise CISO Tom Greco  

Dear Partners, 

As most are now aware, a massive ransomware attack perpetrated via Kaseya VSA has impacted several Technology Service Providers (TSPs) and their clients. Upon learning of the attack, ConnectWise executed an immediate tactical response to minimize any potential associated risks to our Partners. We released a Security Advisory on our Trust Site and via email on Friday evening outlining these actions. We are continuing to monitor the situation and will provide an update if/when necessary based on the potential residual risk to Partners. 
 
Beyond the tactical response, we understand that our Partners may have heightened concerns regarding ConnectWise security as a key vendor supporting your businesses. Further, in light of SolarWinds and this most recent incident, the possibility of supply chain attacks or exploitation of zero-day vulnerabilities is likely topping your list of concerns. 
 
How does ConnectWise view and address these threats? 
 
While I have outlined a few specifics on our security controls below, I also want to invite you to review our newly refreshed and redesigned Trust Center website, which will be the most current source of information about our security practices, SOC2 reports and additional security, compliance, and privacy resources. It also houses our security bulletins, which are now searchable with a variety of filtering options.   

At the top level, our Information Security Program is based upon industry-accepted standards including NIST 800-171, CIS Controls, and ISO 27001.  
 
We expend tremendous effort subjecting our controls to rigorous, independent audits every six months resulting in SOC2 Type 2 reports. These provide third-party attestations that our security controls are designed properly and are operating effectively. In addition to SOC2 certification, ConnectWise is also actively pursuing NIST 800-171 and CMMC compliance. 
 
Additionally, our cloud environments are hosted with world-class providers who possess multiple security certifications including SOC2 Type 2. Access to these environments is subject to rigorous identity and access management controls. Multi-factor authentication is required for all access, privileged or otherwise. Use of privileged accounts is further restricted by conditional and time-bound controls. 
 
All access is also tightly monitored 24/7, employing sophisticated contextual and behavioral methods to detect  anomalies. Our SOC and incident response teams quickly triage and disposition any alerts. 
 
To minimize service interruption, we have established data backup and disaster recovery capabilities within all cloud environments. These include multiple components to minimize the risk of any single point of failure. Access and encryption controls are established to safeguard data back-ups, and all plans are tested and updated regularly. 
 
Our approach to vulnerability management is multi-faceted.  

• We have embraced the Shift Left strategy in our SDLC to detect potential vulnerabilities as early as possible in the development/delivery pipeline.  
• We have improved our secure-by-design efforts including enhanced developer training, updated application security standards, and expanded threat modeling.  
• Our code is also regularly subjected to multiple internal and external penetration tests.  

• To subject our code to even more scrutiny, we have implemented Bug Bounty and Vulnerability Disclosure Programs as well via HackerOne. 

More specific to the supply chain threat, the SolarWinds incident prompted us to execute a threat model against our delivery pipelines in order to identify opportunities for improvement in the associated controls. Areas of focus included, but were not limited to, access and authorization (CI/CD, SCM, and developers), code commits, and configuration management.
 
This is not meant to be an exhaustive view of our efforts in security, but rather to provide some insight into key controls.  We also published resources for MSPs and partners who may have been affected by last week’s events at www.connectwise.com/rapidresponse. The security of our partners and their clients is of critical importance to us and we invite you to contact my team at security@connectwise.com if you have any specific questions or concerns.

Thank you for your continued partnership and stay safe. 

Sincerely, 

Tom Greco 

Chief Information Security Office, ConnectWise  

As you may be aware, Kaseya VSA is experiencing a REvil ransomware attack impacting MSP customers and end customers.   

If your organization utilizes Kaseya VSA, Kaseya has advised that you IMMEDIATELY shut down your VSA server until you receive further notice from them. 

Actions ConnectWise is Taking to Protect Our Partners:

The security of our partners and systems is our top priority. ConnectWise’s Security Operations Center, Network Operations Center, Product and Engineering teams are actively reviewing and monitoring and have thus far found no evidence to suggest that any of our systems are involved or impacted.  

Below are the following actions we are taking to ensure the security of our products and systems:  

• We see no indication of similar attacks, compromises, or suspicious activity associated with ConnectWise products and services. 
• We have temporarily disabled all on-prem and cloud Kaseya and IT Glue integrations into Manage as a precautionary step until more information is available. Our team will share information about re-connecting the access once the all-clear message has been released. 
• Our Security Operations Center (SOC) team has and will continue to carefully monitor the situation. We have taken actions to review the available threat data contained in our SOC monitored systems looking for potentially compromised environments (Fortify Endpoint, Fortify Network, Perch and StratoZen). In addition, we have temporarily removed any exclusions related to the Kaseya agent, and blacklisted the IOCs related to what is currently known of the attack based on our work within the MSP cyber community.  
• The ConnectWise Cyber Research Unit (CRU) is monitoring threat activity from obtained malware samples. We have used these samples to generate and monitor for IoCs (Indicators of Compromise) around this threat. These IoCs are being used to hunt for true positive correlations. 
  • CRU is actively searching for the following IoCs for partners that utilize StratoZen and Perch. Please note that there are additional IoCs that we are currently unable to share. 

1. Multiple C2 domains from JSON malware configuration file which are not being shared at this time.

2. Hashes for the attack structure:

1. agent.exe: 561cffbaba71a6e8cc1cdceda990ead4 (MD5)

2. agent.exe (encrypt payload): SHA15162f14d75e96edb914d1756349d6e11583db0b0

3. mpsvc.dll(sideloaded encryption payload): SHA1 656c4d285ea518d90c1b669b79af475db31e30b1

3. Certificate Signer identity:

1. PB03 TRANSPORT LTD

4. Additional CRU malware sandbox IoCs which cannot yet be publicly shared 

• ConnectWise CRU Event Notifications  
  • The CRU has deployed a new event notification in Perch and StratoZen to alert for any activity around known IoCs from this attack. The ConnectWise SOC is actively monitoring for this alert. 
  • [Windows][CRU] Kaseya Buffalo Jump File Create in "kworking" Directory 
  • Actions deployed in SentinelOne: 
    • All Kaseya exclusions removed from all production SentinelOne consoles. 
    • IOCs of agent.exe and mpsvc.dll blacklisted across all SentinelOne consoles. 
    • IOCs searched across all SentinelOne consoles historical data. 

• We are working and partnering with other vendors to further assist the IT Nation community. 
• ConnectWise Control will offer free temporary STANDARD support licensing available to partners affected by this incident and who do not have a current Control account. Navigate here to sign up for the free license. This will enable impacted partners to maintain connectivity with their client machines during these turbulent times. 

As always, if you ever notice anything that you suspect may be malicious or fraudulent activity within our products, please report them immediately to our InfoSec team at security@connectwise.com.  
 
We will continue to provide updates and information as necessary. 

We want to provide reminders to our partners about email security best practices.  

Phishing remains a significant attack vector fronting attack chains in some very high-profile security incidents.  As such, it is imperative that organizations implement email security controls to prevent impersonation/spoofing of their users and domains.  SPF, DKIM, and DMARC provide a layer of protection against this by working in tandem to authenticate email and helping to ensure that the sender REALLY is who they say they are.   

SPF, DKIM, and DMARC Defined   

• SPF (Sender Policy Framework) is an email validation protocol designed to detect and block email spoofing. It allows mail exchangers to verify that incoming mail from a specific domain comes from an IP Address authorized by that domain’s administrators.  
• DKIM (DomainKeys Identified Mail) utilizes cryptographic signatures by which mail service providers can verify the authenticity of the sender.  
• DMARC (Domain-based Message Authentication, Reporting & Conformance) aligns the SPF and DKIM mechanisms and allows organizations to apply policies regarding unauthorized use of email domains. 

For more information and details on how to setup/configure SPF/DKIM/DMARC, there are several good resources available including the following:   

SPF: https://www.proofpoint.com/us/threat-reference/spf 

DKIM: https://www.proofpoint.com/us/threat-reference/dkim 

DMARC: https://www.proofpoint.com/us/threat-reference/dmarc 

Security is a top priority at ConnectWise. Our primary goal is to provide robust, secure products and services to our partners. We also acknowledge that no technology is perfect, and ConnectWise believes that working with skilled security researchers and partners across the globe is crucial in identifying weaknesses in any technology. If you believe you've found a security issue in our product or service, we encourage you to notify us via our Vulnerability Disclosure Program. We welcome working with you to resolve the issue promptly.  

We are proud to be part of a community that remains equally committed to secure practices. 

We apologize to our partners for the disruption in service last week pertaining to our virtual community. It is now online, and our product and other teams look forward to engaging with you.

Like many ConnectWise experiences (e.g. our University) our virtual community platform leverages SSO to authenticate users and ensure only authorized partners engage in our community. Our SSO mechanism did its job—only allowing verified ConnectWise partners to register, accept the terms and conditions and use the virtual community platform. There was no malicious attack on our SSO capabilities.

Last week, a valued partner (via our VDP and respected admins of the MSPGeek community) raised concern about information our virtual community search was displaying to registered community member partners. Directory search was working as intended in most cases, but a configuration issue was allowing non-registered partners to be returned in a search. This information included "first name", "last name", "company name" (and in some cases, "business title"). Although this information can easily be obtained via other platforms (like LinkedIn), it raised understandable partner concern.  Only 15 registered partner members conducted searches since the community launch, and while we were unable to validate the results of their searches due to a limitation in our vendor’s API, we do know that only 18 non-registered partners "profiles" were viewed by registered partner members as a result of those searches.

We remediated this issue within hours but took the site down pending a full review in accordance with our InfoSec policy. No malicious activity was discovered, no data was lost, and this triggered no data privacy actions in the jurisdictions involved.

Although a common community feature, partners also expressed concern that a registered partner community member could conduct a search by "company name". We understand it is important for partner employees (registered users) to determine how much or how little information is shared with others in the virtual community. Here’s what we did:

• We reconfigured the virtual community to—after authentication—consume only basic information about registered users of the virtual community who accept the terms of service.
• Default settings now limit directory search fields to first name and last name.
• Member directory is “on” for registered partner member viewing to help deliver the experience TSPs expect when joining a virtual community. However, we have set default privacy settings for all registered members such that only their first name, last name (and profile photo where uploaded) will display when being searched for by members who aren’t their approved contacts.
• Registered members may proactively change the privacy settings associated with their user profile to control the level of information that is shared with approved contacts or other members. Partners can find more information about privacy settings in the Virtual Community FAQs.

As a courtesy, we are notifying the 18 individuals mentioned above and are reaching out to the 15 partners who conducted searches to gain their assurance this information will not be used beyond community networking.

Finally, we know it is important to you to hear what we learned from this. Our beta testing (both internal and with partners) in the 30 days prior did not expose this configuration issue. This taught us about extra measures we can and will take in the future; and we have immediately implemented additional multi-layered testing and QC mechanisms to our processes.  

Transparency on all sides benefits our community. We want to thank the partner who reported this, and the partners who collaborated with us on this issue. If you have additional questions about this matter, please contact security@connectwise.com.

Although directory functionality for our virtual community platform was disabled when we launched our community, an issue with our third-party platform’s configuration was discovered. This issue allowed partner first name, last name, and company name (and in some cases, job title) to be returned in the search. We remediated this issue but shut the web site down in an abundance of caution so we could conduct a full assessment in compliance with our InfoSec protocols. To be clear, no malicious activity has been discovered. More specifically, our analysis shows that only partners and ConnectWise employees conducted this search since our community was launched—less than 20 partners searched and many searches were this morning from partners who were helping us test this issue. We have been able to track every search to a legitimate user. We have consulted with our legal counsel, and this has not triggered any GDPR issues. We will share more with our partners when we have more details as our investigation continues.

A potential issue with the virtual community site is being assessed. As a precautionary measure, we have temporarily put the site in maintenance mode while we continue our investigation. To be clear, no malicious activity has been identified. We will update partners shortly.