Another colleague of mine discovered that 4.7.35 failed to work on another project (that I never checked). Perhaps this out-of-memory, while symptomatically similar, is coming from a different source and is not a regression.

Possibly the parser or analyser is getting stuck in a loop somewhere.

Can you run a debug build and try to isolate the source file causing the lock.

Is there a debug build available for download somewhere, or do I need to build my own?

I'm also hitting this problem. It is failing during the Analyse. Unfortunately, by the time I see the problem, it has already hit the exception and jumped out of the actual spot in the code where the problem occurred. I tried to build my own version, but ran in to trouble with getting all the environment variables correct.

Ok, so I found the post about how to build StyleCop and was able to get that up and running. Doing some memory analysis. I'm starting to suspect that this may actually be a Visual Studio issue rather than a StyleCop issue. I don't see any particularly crazy growth of anything within the StyleCop namespaces, however I still get crazy growth of the devenv process memory. When I do an analysis of the dump, it shows the majority of the consumed space as Unknown. Is it possible that there is a bug in the visual studio extension code that is actually the source of the memory leak?

A little more detail. Two main things seem to be contributing to the memory growth. GetSymbols is causing a small growth in memory consumption on each file opened and similarly somewhere in ParseElementContainerBody a little more memory is leaking. Memory consumption doesn't seem to grow in both methods all the time though it could just be file sizes that were too small to be noticeable.

Oh, just another note, I was able to confirm this was also a problem on every version going back to atleast 4.4, including the versions of 4.7 that some had not been able to reproduce the error on.

Ok, more details. I was able to determine that it does appear to actually be a problem with StyleCop after all. To try and work around the problem, I broke out the functionality to a wrapper that calls the StyleCopConsole class directly to process the files I needed to process and I'm still crashing from blowing out the process' memory space.

Running more direct memory profiling has produced some interesting results. Overall there is a slow increase in memory consumption. G2 objects seem to collect without being disposed slowly and overall the memory grows to about 200mb non collecting and 350 to 400mb peak GC.Total memory consumption, however the process at that point is consuming near 2 GB or allocated memory space. I'm trying to investigate if it may be putting stuff in to the large object heap and not clearing it or investigating what else could be causing the drastic discrepancy between private bytes and GC.TotalMemory

Some great investigation work here thank you. If you do identify where the problem is I will fix it straight away, no problem.

I ended up running out of time to spend trying to debug it (for now at least) but I was able to figure out a way to work around it for my purposes. I ended up using the command line client library to build my own engine for processing source files to do the large scale runs I needed. It still was running out of memory, but when I built it as a 64 bit process, it gave me enough expanded memory space that it ran to completion (took 6.2GB of memory space for my solution.)

One other key detail that I did notice along the way though is that as I started addressing issues and rerunning the process, it seems that the memory consumption has been consistently dropping relatively proportionally to the number of violations. Thing is, I've only been dealing with about 1800 to 500 violations, so that's still talking like 3 to 8MB of memory consumption per violation which still seems like an awful lot, unless maybe something like the full lexical graph is not getting garbage collected for some reason as a result of the violation being stored. Average source code file for my solution is only a few hundred to at most a few thousand lines of text.

Oh, one other thought, it is worth noting that after the violation collection was dumped to the output file and all processing finished and collections were emptied, the space did end up being reclaimed. So that also seems to support that the GC is just thinking something large is in use, or perhaps even something is getting carried around that doesn't need to be. Not really sure, the discrepancy between GC.Total and Private Bytes still has me a bit confused as I'm not sure what would be going on to the large object heap (or if there might be other explanations that I haven't considered.)

If you could attach or share the solution I could repro and fix it here.

Hm, perhaps I need to backpedal on my update a little. Doing another run, I'm not seeing the private bytes size drop even after it finished. I just ran a project through and had it take up 3.2gb of ram and it is now staying at 3.2 gb of memory consumption.

Unfortunately, the code base I'm working on is proprietary, so I can't share the solution. I did end up realizing from further testing that I was also mistaken about the link between memory and violations. I forgot that I had altered a filter on input files and that is what caused the apparent reduction. The size in memory is proportional to files read in, not violations.

Any update on a repro for this? I'd love to get it fixed.

I made a memory Profiler run, and found that there are a lot of StyleCop objects using memory.

I think when StyleCop parses Files it analysis non partial Files directly but keeps partial Files in Memory. So If you have a lot of partial Files, the Memory usage grows a lot when you run StyleCop.

So I see two possibilities

1. Don't keep partial Files in Memory during analyzation, massive work to change things, maybe a 3. Analysis path. (Which partial Files belong together).
   
2. Or reduce the Memory Consumption of the StyleCop classes kept in Memory.
   

One candidate would be CodePoint:

namespace StyleCop
{

using System;
using System.Diagnostics.CodeAnalysis;

/// <summary>
/// Describes a point within a code file.
/// </summary>
/// <subcategory>other</subcategory>
public sealed class CodePoint
{

.....

Change to Struct:

//-----------------------------------------------------------------------
// <copyright file="CodePoint.cs">
// MS-PL
// </copyright>
// <license>
// This source code is subject to terms and conditions of the Microsoft
// Public License. A copy of the license can be found in the License.html
// file at the root of this distribution. If you cannot locate the
// Microsoft Public License, please send an email to dlr@microsoft.com.
// By using this source code in any fashion, you are agreeing to be bound
// by the terms of the Microsoft Public License. You must not remove this
// notice, or any other, from this software.
// </license>
//-----------------------------------------------------------------------
namespace StyleCop
{

using System;
using System.Diagnostics.CodeAnalysis;

/// <summary>
/// Describes a point within a code file.
/// </summary>
/// <subcategory>other</subcategory>
public struct CodePoint
{
    #region Private Fields        

    /// <summary>
    /// The index of the first character of the item within the document.
    /// </summary>
    private readonly int index;

    /// <summary>
    /// The index of the first character of the item within the line
    /// that it appears on.
    /// </summary>
    private readonly int indexOnLine;

    /// <summary>
    /// The line number that this item appears on.
    /// </summary>
    private readonly int lineNumber;

    #endregion Private Fields

    #region Public Constructors

    /// <summary>
    /// Initializes a new instance of the CodePoint class.
    /// </summary>
    public CodePoint()
    {
        this.index = 0;
        this.indexOnLine = 0;
        this.lineNumber = 1;
    }

    /// <summary>
    /// Initializes a new instance of the CodePoint class.
    /// </summary>
    /// <param name="index">The index of the first character of the item within the document.</param>
    /// <param name="indexOnLine">The index of the last character of the item within the line
    /// that it appears on.</param>
    /// <param name="lineNumber">The line number that the item appears on.</param>
    [SuppressMessage(
        "Microsoft.Naming",
        "CA1702:CompoundWordsShouldBeCasedCorrectly",
        MessageId = "OnLine",
        Justification = "On Line is two words in this context.")]
    public CodePoint(int index, int indexOnLine, int lineNumber)
    {
        Param.RequireGreaterThanOrEqualToZero(index, "index");
        Param.RequireGreaterThanOrEqualToZero(indexOnLine, "indexOnLine");
        Param.RequireGreaterThanZero(lineNumber, "lineNumber");

        this.index = index;
        this.indexOnLine = indexOnLine;
        this.lineNumber = lineNumber;
    }

    #endregion Public Constructors

    #region Public Properties

    /// <summary>
    /// Gets the index of the first character of the item within the document.
    /// </summary>
    public int Index
    {
        get
        {
            return this.index;
        }
    }

    /// <summary>
    /// Gets the index of the first character of the item within the line
    /// that it appears on.
    /// </summary>
    [SuppressMessage(
        "Microsoft.Naming",
        "CA1702:CompoundWordsShouldBeCasedCorrectly",
        MessageId = "OnLine",
        Justification = "On Line is two words in this context.")]
    public int IndexOnLine
    {
        get
        {
            return this.indexOnLine;
        }
    }

    /// <summary>
    /// Gets the line number that this item appears on.
    /// </summary>
    public int LineNumber
    {
        get
        {
            return this.lineNumber;
        }
    }

    #endregion Public Properties

    #region Public Static Methods

    /// <summary>
    /// Joins the two given points.
    /// </summary>
    /// <param name="point1">The first point to join.</param>
    /// <param name="point2">The second point to join.</param>
    /// <returns>Returns the joined <see cref="CodePoint"/>.</returns>
    public static CodePoint Join(CodePoint point1, CodePoint point2)
    {
        if (point1.lineNumber == 0 && point1.indexOnLine == 0 && point1.index == 0)
        {
            return point2;
        }
        else if (point2.lineNumber == 0 && point2.indexOnLine == 0 && point2.index == 0)
        {
            return point1;
        }
        else
        {
            // Figure out which IndexOnLine to use.
            int indexOnLine;
            if (point1.LineNumber == point2.LineNumber)
            {
                indexOnLine = Math.Min(point1.IndexOnLine, point2.IndexOnLine);
            }
            else if (point1.LineNumber < point2.LineNumber)
            {
                indexOnLine = point1.IndexOnLine;
            }
            else
            {
                indexOnLine = point2.IndexOnLine;
            }

            return new CodePoint(
                Math.Min(point1.Index, point2.Index),
                indexOnLine,
                Math.Min(point1.LineNumber, point2.LineNumber));
        }
    }

    #endregion Public Static Methods
}

}

The semantics shouldn't change because In Code Point all fields are already readonly and it is sealed. (Because a struct is in principle a immutable object. Like this class).

See this for an explanation:
http://www.dotnetperls.com/struct

I made some more tests and discovered something very intresting, StyleCop is included in the Msbuild with the nuget msbuild StyleCop task. I separated StyleCop away.

with following MsBuild Task.

<UsingTask AssemblyFile="$(MSBuildProjectDirectory)\packages\StyleCop.MSBuild.4.7.42.1\tools\StyleCop.dll" TaskName="StyleCopTask"/>

<PropertyGroup>
    <FolderToAnalyse>$(MSBuildProjectDirectory)</FolderToAnalyse>
    <ProjectPath>$(MSBuildProjectDirectory)\Famc.sln</ProjectPath>
    <StyleCopForceFullAnalysis>true</StyleCopForceFullAnalysis>
    <DefineConstants>DEBUG;TRACE</DefineConstants>
    <StyleCopTreatErrorsAsWarnings>true</StyleCopTreatErrorsAsWarnings>
    <StyleCopCacheResults>true</StyleCopCacheResults>
    <StyleCopOverrideSettingsFile>$(MSBuildProjectDirectory)\Settings.StyleCop</StyleCopOverrideSettingsFile>
    <StyleCopOutputFile>StyleCopViolations.xml</StyleCopOutputFile>
    <StyleCopMaxViolationCount>1000</StyleCopMaxViolationCount>
</PropertyGroup>

<Target Name="StyleCop">

    <CreateItem Include="$(FolderToAnalyse)\**\*.cs" Exclude="$(FolderToAnalyse)\**\*Designer.cs">
        <Output TaskParameter="Include" ItemName="StyleCopFiles"/>
    </CreateItem>

    <StyleCopTask
      ProjectFullPath="$(ProjectPath)"
      SourceFiles="@(StyleCopFiles)"
      AdditionalAddinPaths="$(MSBuildProjectDirectory)\packages\StyleCop.MSBuild.4.7.42.1\tools\"
      ForceFullAnalysis="$(StyleCopForceFullAnalysis)"
      DefineConstants="$(DefineConstants)"
      TreatErrorsAsWarnings="$(StyleCopTreatErrorsAsWarnings)"
      CacheResults="$(StyleCopCacheResults)"
      OverrideSettingsFile="$(StyleCopOverrideSettingsFile)"
      OutputFile="$(StyleCopOutputFile)"
      MaxViolationCount="$(StyleCopMaxViolationCount)"
        />

</Target>

This finished very quickly with 800Mb Memory usage, If I removed the exclude:
Exclude="$(FolderToAnalyse)***Designer.cs"

It grow near 4 GB of memory usage and crashed. So I think even if the .Designer files are excluded per StyleCop setting they are analysed and make the Memory Usage grow very quickly. Some Designer Files are rather big one is more than 4MB in Code (Typed DataSet, And Active Reports Designer files)

Thanks for the help. Looking at this again now.

I can verify that it looks partial class related. While I haven't had a chance to do additional testing after seeing the post about partial classes, I can confirm that the project that was experiencing the issues makes MASSIVE use of partial classes. Like, every single class is a partial class and there are several hundred of them.