import subprocess import os import filecmp import glob import sys # Import sys for platform detection import time import shutil import re # Import re for regex parsing on macOS # --- Configuration --- CXX = "g++" CXXFLAGS = ["-Wall", "-O2", "-std=c++11"] EXECUTABLE = "./nytrends.exe" SOURCE_FILES_PATTERN = "*.cpp" INPUT_DIR = "inputs" EXPECTED_OUTPUT_DIR = "outputs" TEMP_OUTPUT_FILE = "output_unit_test.txt" TEST_TIMEOUT = 120 # Configuration for memory measurement MEASURE_MEMORY = True # Master switch TIME_COMMAND = "/usr/bin/time" # --- Platform Specific Time Config --- TIME_COMMAND_MODE = None # Will be 'linux' or 'macos' or None LINUX_TIME_FORMAT = "%M" # Format specifier for Max RSS (KB) on Linux LINUX_TIME_OUTPUT_FILE = "time_mem_output.tmp" # Temp file for Linux time output MACOS_MEM_REGEX = re.compile(r"^\s*(\d+)\s+maximum resident set size", re.IGNORECASE | re.MULTILINE) # Configuration for suppressing program output SUPPRESS_PROGRAM_OUTPUT = True # ANSI Color Codes # ... (colors remain the same) ... COLOR_GREEN = '\033[92m' COLOR_RED = '\033[91m' COLOR_YELLOW = '\033[93m' COLOR_BLUE = '\033[94m' COLOR_RESET = '\033[0m' # --- Helper Functions --- def print_color(text, color): """Prints text in a specified color.""" print(f"{color}{text}{COLOR_RESET}") def check_time_command(): """ Check if /usr/bin/time command exists and is usable for memory measurement based on the OS. Sets TIME_COMMAND_MODE. Returns True if usable, False otherwise. """ global TIME_COMMAND_MODE if not shutil.which(TIME_COMMAND): print_color(f"Warning: '{TIME_COMMAND}' not found. Memory measurement disabled.", COLOR_YELLOW) TIME_COMMAND_MODE = None return False platform = sys.platform test_command = [] capture_stderr = False if platform.startswith("linux"): test_command = [TIME_COMMAND, '-f', LINUX_TIME_FORMAT, 'true'] capture_stderr = False # Output goes to stdout/stderr, just check exit code TIME_COMMAND_MODE = "linux" print(f"Detected Linux platform. Testing {TIME_COMMAND} with '-f {LINUX_TIME_FORMAT}'...") elif platform == "darwin": # macOS test_command = [TIME_COMMAND, '-l', 'true'] capture_stderr = True # Need to capture stderr to check output format TIME_COMMAND_MODE = "macos" print(f"Detected macOS platform. Testing {TIME_COMMAND} with '-l'...") else: print_color(f"Warning: Unsupported platform '{platform}' for memory measurement. Disabled.", COLOR_YELLOW) TIME_COMMAND_MODE = None return False try: # Run test command process = subprocess.run(test_command, capture_output=True, # Capture both stdout/stderr text=True, check=True, # Raise exception on non-zero exit timeout=3) # Additional check for macOS output format if TIME_COMMAND_MODE == "macos": if MACOS_MEM_REGEX.search(process.stderr): print_color(f"Memory measurement enabled using '{TIME_COMMAND} -l'.", COLOR_GREEN) return True # Format looks okay else: print_color(f"Warning: '{TIME_COMMAND} -l' output format not recognized (missing 'maximum resident set size'). Memory measurement disabled.", COLOR_YELLOW) TIME_COMMAND_MODE = None return False else: # Linux check passed if check=True didn't raise exception print_color(f"Memory measurement enabled using '{TIME_COMMAND} -f {LINUX_TIME_FORMAT}'.", COLOR_GREEN) return True except subprocess.CalledProcessError as e: # This is where the original macOS error occurred print_color(f"Warning: {TIME_COMMAND} test command failed (exit code {e.returncode}). Memory measurement disabled.", COLOR_YELLOW) if e.stderr: print(f"Stderr:\n{e.stderr}") TIME_COMMAND_MODE = None return False except FileNotFoundError: # Should have been caught by shutil.which, but belt-and-suspenders print_color(f"Warning: '{TIME_COMMAND}' not found during test run. Memory measurement disabled.", COLOR_YELLOW) TIME_COMMAND_MODE = None return False except Exception as e: print_color(f"Warning: An unexpected error occurred while testing {TIME_COMMAND}. Memory measurement disabled. Error: {e}", COLOR_YELLOW) TIME_COMMAND_MODE = None return False # --- compile_program() remains the same --- def compile_program(): """Compiles the C++ source files.""" print_color(f"--- Starting Compilation ---", COLOR_BLUE) source_files = glob.glob(SOURCE_FILES_PATTERN) if not source_files: print_color(f"Error: No source files found matching pattern '{SOURCE_FILES_PATTERN}'.", COLOR_RED) return False compile_command = [CXX] + CXXFLAGS + ["-o", os.path.basename(EXECUTABLE)] + source_files command_str = " ".join(compile_command) print(f"Running: {command_str}") try: start_time = time.perf_counter() process = subprocess.run(compile_command, check=False, capture_output=True, text=True) end_time = time.perf_counter() duration = end_time - start_time if process.returncode == 0: print_color(f"Compilation successful (took {duration:.3f}s).", COLOR_GREEN) if process.stderr: print_color("Compiler Warnings/Messages:", COLOR_YELLOW) print(process.stderr) return True else: print_color(f"Compilation failed with exit code {process.returncode} (took {duration:.3f}s).", COLOR_RED) print_color("Compiler Error Output:", COLOR_RED) print(process.stderr if process.stderr else "(No compiler error output captured)") return False except FileNotFoundError: print_color(f"Error: Compiler '{CXX}' not found.", COLOR_RED) return False except Exception as e: print_color(f"An unexpected error occurred during compilation: {e}", COLOR_RED) return False def run_test(test_name, input_file, expected_output_file, argument): """ Runs test, measures time/memory (platform-specific), suppresses output. Returns: tuple (passed: bool, reason: str, duration: float | None, memory_kb: int | None) """ global MEASURE_MEMORY, TIME_COMMAND_MODE # Access potentially updated flags print_color(f"--- Running {test_name} ---", COLOR_BLUE) duration = None memory_kb = None captured_stderr_for_mem = None # Store stderr specifically for macos parsing # Prerequisite checks if not os.path.exists(input_file): return False, "Input file missing", None, None if not os.path.exists(expected_output_file): return False, "Expected output file missing", None, None if not os.path.exists(EXECUTABLE): return False, "Executable not found", None, None # --- Command Construction & subprocess args --- base_command = [EXECUTABLE, input_file, TEMP_OUTPUT_FILE, argument] run_command = [] subprocess_kwargs = { # Base arguments for subprocess.run "check": False, "timeout": TEST_TIMEOUT } if MEASURE_MEMORY and TIME_COMMAND_MODE: # Check both desire and capability if TIME_COMMAND_MODE == "linux": run_command = [TIME_COMMAND, '-f', LINUX_TIME_FORMAT, '-o', LINUX_TIME_OUTPUT_FILE] + base_command if os.path.exists(LINUX_TIME_OUTPUT_FILE): try: os.remove(LINUX_TIME_OUTPUT_FILE) except OSError: pass # For Linux, memory info goes to file, handle stdout/stderr normally based on suppression subprocess_kwargs["stdout"] = subprocess.DEVNULL if SUPPRESS_PROGRAM_OUTPUT else None subprocess_kwargs["stderr"] = subprocess.DEVNULL if SUPPRESS_PROGRAM_OUTPUT else None elif TIME_COMMAND_MODE == "macos": run_command = [TIME_COMMAND, '-l'] + base_command # On macOS, need to capture stderr for parsing memory, stdout handles suppression subprocess_kwargs["stdout"] = subprocess.DEVNULL if SUPPRESS_PROGRAM_OUTPUT else None subprocess_kwargs["stderr"] = subprocess.PIPE # Capture stderr for parsing subprocess_kwargs["text"] = True # Decode captured stderr else: # Not measuring memory or platform unsupported run_command = base_command subprocess_kwargs["stdout"] = subprocess.DEVNULL if SUPPRESS_PROGRAM_OUTPUT else None subprocess_kwargs["stderr"] = subprocess.DEVNULL if SUPPRESS_PROGRAM_OUTPUT else None command_str = " ".join(run_command) print(f"Executing: {command_str}") # --- Execution and Measurement --- if os.path.exists(TEMP_OUTPUT_FILE): try: os.remove(TEMP_OUTPUT_FILE) except OSError as e: print_color(f"Warning: Could not remove {TEMP_OUTPUT_FILE}: {e}", COLOR_YELLOW) try: start_time = time.perf_counter() process = subprocess.run(run_command, **subprocess_kwargs) end_time = time.perf_counter() duration = end_time - start_time print(f"Execution Time: {duration:.3f} seconds") # --- Process Memory Output (Platform Specific) --- if MEASURE_MEMORY and TIME_COMMAND_MODE: if TIME_COMMAND_MODE == "linux": if os.path.exists(LINUX_TIME_OUTPUT_FILE): try: with open(LINUX_TIME_OUTPUT_FILE, 'r') as f_time: mem_str = f_time.read().strip() if mem_str: memory_kb = int(mem_str) # Already in KB print(f"Peak Memory Usage: {memory_kb} KB") else: print_color(f"Warning: {LINUX_TIME_OUTPUT_FILE} was empty.", COLOR_YELLOW) except (ValueError, IOError) as e: print_color(f"Warning: Could not parse memory (Linux) from {LINUX_TIME_OUTPUT_FILE}: {e}", COLOR_YELLOW) finally: try: os.remove(LINUX_TIME_OUTPUT_FILE) except OSError: pass else: print_color(f"Warning: {LINUX_TIME_OUTPUT_FILE} was not created.", COLOR_YELLOW) elif TIME_COMMAND_MODE == "macos": # Parse memory from captured stderr (process.stderr) if process.stderr: match = MACOS_MEM_REGEX.search(process.stderr) if match: try: mem_bytes = int(match.group(1)) memory_kb = mem_bytes // 1024 # Convert Bytes to KB print(f"Peak Memory Usage: {memory_kb} KB ({mem_bytes} Bytes)") except (ValueError, IndexError): print_color(f"Warning: Could not parse memory value (macOS) from captured output.", COLOR_YELLOW) # Optional: print process.stderr here for debugging # print(f"--- time -l stderr ---\n{process.stderr}\n----------------------") else: print_color(f"Warning: 'maximum resident set size' not found in 'time -l' output (macOS).", COLOR_YELLOW) # Optional: print process.stderr here for debugging # print(f"--- time -l stderr ---\n{process.stderr}\n----------------------") else: print_color(f"Warning: No stderr captured from 'time -l' (macOS).", COLOR_YELLOW) # --- Check Program Result --- if process.returncode != 0: print_color(f"Test failed: Program exited with non-zero status {process.returncode}.", COLOR_RED) # Note: program's own stderr might be in process.stderr ONLY if not suppressed AND on macOS # It's generally hidden now by design. return False, "Runtime error", duration, memory_kb if not os.path.exists(TEMP_OUTPUT_FILE): print_color(f"Test failed: Program finished successfully but did not create '{TEMP_OUTPUT_FILE}'.", COLOR_RED) return False, "Output file not created", duration, memory_kb # --- Compare Output File --- if filecmp.cmp(TEMP_OUTPUT_FILE, expected_output_file, shallow=False): print_color(f"Test Result: PASSED", COLOR_GREEN) return True, "Passed", duration, memory_kb else: # ... (diff printing remains the same) ... print_color(f"Test Result: FAILED - Output mismatch.", COLOR_RED) print_color(f" Expected: {expected_output_file}", COLOR_YELLOW) print_color(f" Actual: {TEMP_OUTPUT_FILE}", COLOR_YELLOW) try: diff_proc = subprocess.run(['diff', '-u', expected_output_file, TEMP_OUTPUT_FILE], capture_output=True, text=True) print_color("--- Diff ---", COLOR_YELLOW) print(diff_proc.stdout if diff_proc.stdout else "(No differences found by diff, might be whitespace or encoding issues)") print_color("------------", COLOR_YELLOW) except FileNotFoundError: print_color("Could not run 'diff' command.", COLOR_YELLOW) except Exception as diff_e: print_color(f"Error running diff: {diff_e}", COLOR_YELLOW) return False, "Output mismatch", duration, memory_kb # --- Exception Handling --- except subprocess.TimeoutExpired: end_time = time.perf_counter() duration = end_time - start_time print_color(f"Test failed: Program timed out after {duration:.3f}s (limit: {TEST_TIMEOUT}s).", COLOR_RED) # Attempt to parse memory ONLY if macOS and stderr might have been partially captured (unlikely but possible) if MEASURE_MEMORY and TIME_COMMAND_MODE == "macos" and process and process.stderr: match = MACOS_MEM_REGEX.search(process.stderr) if match: try: memory_kb = int(match.group(1)) // 1024 except: memory_kb = None # Ignore parsing errors on timeout # Clean up Linux temp file if it exists if MEASURE_MEMORY and TIME_COMMAND_MODE == "linux" and os.path.exists(LINUX_TIME_OUTPUT_FILE): try: os.remove(LINUX_TIME_OUTPUT_FILE) except OSError: pass return False, "Timeout", duration, memory_kb except Exception as e: print_color(f"An unexpected error occurred during test execution: {e}", COLOR_RED) # Clean up Linux temp file if it exists if MEASURE_MEMORY and TIME_COMMAND_MODE == "linux" and os.path.exists(LINUX_TIME_OUTPUT_FILE): try: os.remove(LINUX_TIME_OUTPUT_FILE) except OSError: pass return False, f"Execution exception: {e}", None, None finally: # General cleanup (Linux temp file might still exist if parsing failed) if MEASURE_MEMORY and TIME_COMMAND_MODE == "linux" and os.path.exists(LINUX_TIME_OUTPUT_FILE): try: os.remove(LINUX_TIME_OUTPUT_FILE) except OSError: pass # --- Main Execution --- if __name__ == "__main__": # 0. Check if memory measurement is desired AND possible user_wants_memory_measurement = MEASURE_MEMORY if user_wants_memory_measurement: can_actually_measure = check_time_command() MEASURE_MEMORY = can_actually_measure # Update based on check else: MEASURE_MEMORY = False print_color("Memory measurement explicitly disabled by configuration.", COLOR_YELLOW) if SUPPRESS_PROGRAM_OUTPUT: print_color("Program stdout/stderr will be suppressed during tests.", COLOR_BLUE) # 1. Compile if not compile_program(): print_color("\nCompilation failed. Aborting tests.", COLOR_RED) sys.exit(1) # 2. Define Test Cases # ... (test_bases and arguments_to_test remain the same) ... test_bases = [ ("1", "tiny1"), ("2", "tiny2"), ("3", "small1"), ("4", "small2"), ("5", "medium1"), ("6", "medium2"), ("7", "large1"), ("8", "large2"), ("9", "large3"), ("10", "large4"), ("11", "large5"), ("12", "large6"), ("13", "large7"), ("14", "large8"), ("15", "large9"), ] arguments_to_test = ["hashtag", "sound"] results = {"passed": 0, "failed": 0, "skipped": 0} failed_tests = [] test_durations = [] test_memory_usages = [] # 3. Run Tests print_color("\n--- Starting Test Execution ---", COLOR_BLUE) total_start_time = time.perf_counter() for id_prefix, base_name in test_bases: for i, argument in enumerate(arguments_to_test, 1): # ... (construct test names/paths) ... test_id = f"{id_prefix}.{i}" test_name = f"Test Case {test_id}: input {base_name}, {argument}" input_filename = os.path.join(INPUT_DIR, f"input_{base_name}.json") expected_output_filename = os.path.join(EXPECTED_OUTPUT_DIR, f"output_{base_name}_{argument}.txt") passed, reason, duration, memory_kb = run_test(test_name, input_filename, expected_output_filename, argument) # ... (Update results logic remains the same, relies on memory_kb being None if not measured) ... if passed: results["passed"] += 1 if duration is not None: test_durations.append(duration) if MEASURE_MEMORY and memory_kb is not None: test_memory_usages.append(memory_kb) elif reason in ["Input file missing", "Expected output file missing", "Executable not found"]: results["skipped"] += 1 else: results["failed"] += 1 duration_str = f" ({duration:.3f}s)" if duration is not None else "" mem_str = f", {memory_kb} KB" if MEASURE_MEMORY and memory_kb is not None else "" failed_tests.append(f"{test_name} ({reason}{duration_str}{mem_str})") print("-" * 40) total_end_time = time.perf_counter() total_test_suite_duration = total_end_time - total_start_time # 4. Clean up # ... (same cleanup logic) ... print_color("--- Cleaning Up ---", COLOR_BLUE) if os.path.exists(TEMP_OUTPUT_FILE): try: os.remove(TEMP_OUTPUT_FILE) print(f"Removed temporary output file: {TEMP_OUTPUT_FILE}") except OSError as e: print_color(f"Warning: Could not remove {TEMP_OUTPUT_FILE}: {e}", COLOR_YELLOW) if os.path.exists(EXECUTABLE): try: os.remove(EXECUTABLE) print(f"Removed executable: {EXECUTABLE}") except OSError as e: print_color(f"Warning: Could not remove {EXECUTABLE}: {e}", COLOR_YELLOW) # 5. Print Summary # ... (summary printing logic remains the same) ... # Note: Memory summary section only appears if MEASURE_MEMORY is True at the end. print_color("\n--- Test Summary ---", COLOR_BLUE) print_color(f"Passed: {results['passed']}", COLOR_GREEN) print_color(f"Failed: {results['failed']}", COLOR_RED if results['failed'] > 0 else COLOR_GREEN) print_color(f"Skipped: {results['skipped']}", COLOR_YELLOW if results['skipped'] > 0 else COLOR_GREEN) total_run = results['passed'] + results['failed'] total_defined = total_run + results['skipped'] print(f"Total Tests Defined: {total_defined}") print(f"Total Tests Run: {total_run}") print(f"Total Test Suite Execution Time: {total_test_suite_duration:.3f}s") # Performance Summary if test_durations: # ... (same calculation and printing) ... total_passed_time = sum(test_durations) avg_time = total_passed_time / len(test_durations) max_time = max(test_durations) min_time = min(test_durations) print("\n--- Performance Summary (Passed Tests) ---") print(f"Total execution time (passed tests): {total_passed_time:.3f}s") print(f"Average execution time per test: {avg_time:.3f}s") print(f"Fastest test execution time: {min_time:.3f}s") print(f"Slowest test execution time: {max_time:.3f}s") # Memory Summary if MEASURE_MEMORY: # Check final flag state if test_memory_usages: # ... (same calculation and printing) ... total_mem_kb = sum(test_memory_usages) avg_mem_kb = total_mem_kb / len(test_memory_usages) max_mem_kb = max(test_memory_usages) min_mem_kb = min(test_memory_usages) total_mem_mb = total_mem_kb / 1024 total_mem_gb = total_mem_mb / 1024 if total_mem_gb > 1: total_mem_str = f"{total_mem_gb:.2f} GB" elif total_mem_mb > 1: total_mem_str = f"{total_mem_mb:.2f} MB" else: total_mem_str = f"{total_mem_kb} KB" print("\n--- Memory Usage Summary (Passed Tests) ---") print(f"Cumulative peak memory (passed tests): {total_mem_str} ({total_mem_kb} KB)") print(f"Average peak memory per test: {avg_mem_kb:.1f} KB") print(f"Lowest peak memory usage: {min_mem_kb} KB") print(f"Highest peak memory usage: {max_mem_kb} KB") else: print("\n--- Memory Usage Summary (Passed Tests) ---") print("(No memory usage data collected for passed tests - check warnings)") # Final Result if failed_tests: print_color("\n--- Failed Test Cases ---", COLOR_RED) for test in failed_tests: print(f" - {test}") sys.exit(1) # ... (rest of exit logic remains the same) ... elif results['passed'] == 0 and results['skipped'] == total_defined: print_color("\nWarning: No tests were executed (all skipped).", COLOR_YELLOW) sys.exit(0) elif results['passed'] > 0 : print_color("\nAll executed tests passed successfully!", COLOR_GREEN) sys.exit(0) else: print_color("\nNo tests passed.", COLOR_YELLOW) sys.exit(1 if results['failed'] > 0 else 0)