Inclusion Criteria: * Age \>= 18 years * Documented CD20+ mature B-cell neoplasm according to World Health Organization (WHO) classification (Swerdlow et al., 2016) as one of the following: * Diffuse large B-cell lymphoma not otherwise specified (NOS) including * Transformed lymphoma * Richter's transformation * Germinal center B-cell type * Activated B-cell type * High-grade B-cell lymphoma (HGBCL), NOS * Primary mediastinal (thymic) large B-cell lymphoma * Patients with "double-hit" or "triple-hit" diffuse large B-cell lymphoma (DLBCL) (technically as HGBCL, with MYC and BCL2 and/or BCL6 rearrangements) * Follicular lymphoma 3B * T-cell/histiocyte-rich large B cell lymphoma * Large B-cell lymphoma with IRF4 rearrangement * Primary cutaneous DLBCL, leg type * Epstein-Barr virus (EBV) positive DLBCL, NOS * DLBCL associated with chronic inflammation * Intravascular large B-cell lymphoma * ALK positive large B-cell lymphoma * Relapsed, progressive and/or refractory disease (Cheson et al., 2007) following treatment with an anti-CD20 monoclonal antibody (e.g., rituximab) in combination with chemotherapy * Measurable disease as defined below: * Fluorodeoxyglucose (FDG)-avid lymphomas: Measurable disease with computerized tomography (CT) (or magnetic resonance imaging \[MRI\]) scan with involvement of 2 or more clearly demarcated lesions/nodes with a long axis \> 1.5 cm and short axis \> 1.0 cm (or 1 clearly demarcated lesion/node with a long axis \> 2.0 cm and short axis \>= 1.0 cm) AND FDG positron emission tomography (PET) scan that demonstrates positive lesion(s) compatible with CT (or MRI) defined anatomical tumor sites * FDG-nonavid lymphomas: Measurable disease with CT (or MRI) scan with involvement of 2 or more clearly demarcated lesions/nodes with a long axis \> 1.5 cm and short axis \> 1.0 cm or 1 clearly demarcated lesion/node with a long axis \> 2.0 cm and short axis \>= 1.0 cm. * Eastern Cooperative Oncology Group (ECOG) Performance Status (PS) 0 or 1 * \>= 4 weeks from last dose of anti-CD20 targeting therapy * \>= 12 weeks post chimeric antigen receptor (CAR) T-cell therapy * Resolution of all adverse events due to prior therapy to =\< Grade 1 or baseline NOTE: Patients with =\< Grade 2 neuropathy may be eligible. Patients with endocrine-related adverse events (AEs) Grade =\< 2 requiring treatment or hormone replacement may be eligible * If receiving glucocorticoid treatment at screening, treatment must be tapered down and administered with a maximum of 10 mg daily in the last 14 days prior to registration * Absolute neutrophil count (ANC) \>= 500/mm\^3; growth factor support allowed in case of bone marrow involvement (obtained =\< 7 days prior to registration) * Absolute lymphocyte count \>= 200/mm\^3 (obtained =\< 7 days prior to registration) * Platelet count \>= 75,000/mm\^3 (obtained =\< 7 days prior to registration) * Hemoglobin \>= 8.0 g/dL (obtained =\< 7 days prior to registration) * International normalized ratio (INR) or activated partial thromboplastin time (aPTT) =\< 1.5 × upper limit of normal (ULN) unless participant is receiving anticoagulant therapy as long as prothrombin time (PT) or aPTT is within therapeutic range of intended use of anticoagulants (obtained =\< 7 days prior to registration) * Total bilirubin =\< 1.5 x upper limit of normal (ULN), unless due to Gilbert's disease (direct bilirubin \[bili\] =\< ULN) (obtained =\< 7 days prior to registration) * Aspartate transaminase \[AST/serum glutamic oxaloacetic transaminase (SGOT)\] and alanine transaminase \[ALT/serum glutamic pyruvic transaminase (SGPT)\] =\< 2.5 x ULN (obtained =\< 7 days prior to registration) * Calculated creatinine clearance \>=30 mL/min using the Cockcroft-Gault formula (obtained =\< 7 days prior to registration) * Provide informed written consent * Negative pregnancy test done =\< 3 days prior to registration, for persons of childbearing potential only * Female of childbearing must agree to use a highly effective method of contraception during the treatment and for 120 days after the last dose of study treatment * Male participants with female partners of childbearing potential must agree to refrain from donating sperm and one of the conception methods during the treatment and for 120 days after last dose study treatment * Willing to return to the enrolling institution for follow-up (during the active monitoring phase of the study) * Willing to provide mandatory tissue and blood samples for correlative research purposes Exclusion Criteria: * Primary central nervous system (CNS) lymphoma or known CNS involvement by lymphoma at screening as confirmed by magnetic resonance imaging (MRI)/computed tomography (CT) scan (brain) and, if clinically indicated, by lumbar puncture * Known past or current malignancy other than inclusion diagnosis, except for: * Cervical carcinoma of Stage 1B or less * Non-invasive basal cell or squamous cell skin carcinoma * Non-invasive, superficial bladder cancer * Prostate cancer with a current prostate specific antigen (PSA) level \< 0.1 ng/mL * Any curable cancer with a complete response (CR) of \> 2 years duration * Received \< 2 prior systemic anti-cancer therapy including investigational agents =\< 4 weeks or =\< 5 half-lives, whichever is shorter, prior to registration * Received prior therapy with an anti-PD-1, anti-PD-L1, or anti-PD-L2 agent or with an agent directed to another stimulatory or co-inhibitory T-cell receptor (e.g., CTLA-4, OX-40, CD137) =\< 4 weeks prior to registration * Known clinically significant cardiac disease, including: * Onset of unstable angina pectoris within 6 months of signing informed consent form (ICF) * Acute myocardial infarction within 6 months of signing ICF * Congestive heart failure (grade III or IV as classified by the New York Heart Association and/or known decrease ejection fraction of \< 45%) * Chronic ongoing infectious diseases (except hepatitis B or hepatitis C) requiring treatment (excluding prophylactic treatment) at the time of enrollment or =\< the previous 2 weeks * Confirmed history or current autoimmune disease or other diseases resulting in permanent immunosuppression or requiring permanent immunosuppressive therapy or primary immunodeficiency disorder. Low-dose steroids (=\< 10 mg daily of prednisone equivalent) is allowed * Seizure disorder requiring therapy (such as steroids or anti-epileptics) * Autologous hematopoietic stem cell transplant (HSCT) =\< 100 days prior or any prior allogeneic HSCT or solid organ transplantation * Known human immunodeficiency virus (HIV) infection * Exposed to live or live attenuated vaccine =\< 4 weeks prior to registration * Any of the following because this study involves an investigational agent whose genotoxic, mutagenic and teratogenic effects on the developing fetus and newborn are unknown: * Pregnant persons * Nursing persons * Persons of childbearing potential who are unwilling to employ adequate contraception * Patient has any condition for which, in the opinion of the investigator, participation would not be in the best interest of the patient (e.g., compromise the well-being) or that could prevent, limit, or confound the protocol-specified assessments * Co-morbid systemic illnesses or other severe concurrent disease which, in the judgment of the investigator, would make the patient inappropriate for entry into this study or interfere significantly with the proper assessment of safety and toxicity of the prescribed regimens * Uncontrolled intercurrent illness including, but not limited to: * ongoing or active infection * uncontrolled infection requiring ongoing antibiotics * symptomatic congestive heart failure * unstable angina pectoris * cardiac arrhythmia * or psychiatric illness/social situations that would limit compliance with study requirements * known substance abuse disorder * Known hypersensitivity to pembrolizumab * Major surgery other than diagnostic surgery =\< 4 weeks prior to registration * Prior radiation therapy =\< 2 weeks prior to registration or who has not recovered from all radiation-related toxicities, not require corticosteroids, and not have had radiation pneumonitis. Note: A 1-week washout is permitted for palliative radiation (=\< 2 weeks of radiotherapy) to non-CNS disease * Active autoimmune disease such as Crohn's disease, rheumatoid arthritis, Sjogren's disease, systemic lupus erythematosus, or similar conditions requiring systemic treatment =\< the past 3 months or a documented history of clinically severe autoimmune disease/syndrome difficult to control in the past. EXCEPTIONS: * Vitiligo or resolved childhood asthma/atopy * Intermittent use of bronchodilators or local steroid injections * Hypothyroidism stable on hormone replacement, * Diabetes stable with current management * History of positive Coombs test but no evidence of hemolysis * Psoriasis not requiring systemic treatment * Conditions not expected to recur in the absence of an external trigger * Has a known history of hepatitis B (defined as hepatitis B surface antigen \[HBsAg\] reactive) or known active hepatitis C virus (HCV) (defined as HCV ribonucleic acid \[RNA\] is detected) infection * Prior anti CD47 therapy * Active use of anticoagulant like warfarin. Use of low molecular weight heparin and factor Xa inhibitors will be permitted on case by case basis. There will be no restriction for daily aspirin \< 81mg daily