Since macOS Big Sur 11.3.1, this option no longer affects MarvinSketch, that is even when the value of the Prefer tabs is set to always, MarvinSketch runs without problems.

The application freezing issue is caused by a change in tab behavior and it is present in macOS versions 11.3 and earlier. Newly opened dialogs are handled as tabs by default now. 

The issue can be solved by changing the tab preferences of the system.

To do this, follow the steps below:

  1. Open System Preferences.
  2. Navigate to General.
  3. Change Prefer tabs to never.